Emmanuel Pilliat

Alexandra Carpentier

Nicolas Verzelen

Optimal Permutation Estimation in Crowd-Sourcing problems

Motivated by crowd-sourcing applications, we consider a model where we have partial observations from a bivariate isotonic n × d matrix with an unknown permutation π * acting on its rows. Focusing on the twin problems of recovering the permutation π * and estimating the unknown matrix, we introduce a polynomial-time procedure achieving the minimax risk for these two problems, this for all possible values of n, d, and all possible sampling efforts. Along the way, we establish that, in some regimes, recovering the unknown permutation π * is considerably simpler than estimating the matrix.

Introduction

We consider a crowd-sourcing problem with n experts and d questions. For an unknown matrix M , M i,j ∈ [0, 1] stands for the ability of expert i at question j. For the purpose of calibrating the model, we receive partial and noisy observations of the matrix M and our goal is to rank the experts according to their ability. Earlier models in crowd-labelling problems or in the related problems of pairwise comparisons typically assumed that the matrix M belongs to a parametric model [START_REF] Bradley | Rank analysis of incomplete block designs: I. the method of paired comparisons[END_REF][START_REF] Luce | Individual choice behavior: A theoretical analysis[END_REF][START_REF] Thurstone | A law of comparative judgment[END_REF][START_REF] Dawid | Maximum likelihood estimation of observer error-rates using the em algorithm[END_REF][START_REF] Braverman | Noisy sorting without resampling[END_REF], a prominent example being Bradley-Luce-Terry model. While there has been significant progress in this direction, such models do not tend to fit well real-world data [START_REF] Mclaughlin | Stochastic transitivity and cancellation of preferences between bitter-sweet solutions[END_REF][START_REF] Ballinger | Decisions, error and heterogeneity[END_REF].

To address this issue, there has been a recent interest in the class of permutation-based models [START_REF] Chatterjee | Matrix estimation by universal singular value thresholding[END_REF][START_REF] Shah | Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence[END_REF][START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF][START_REF] Chatterjee | Estimation in tournaments and graphs under monotonicity constraints[END_REF][START_REF] Flammarion | Optimal rates of statistical seriation[END_REF][START_REF] Pananjady | Isotonic regression with unknown permutations: Statistics, computation and adaptation[END_REF][START_REF] Shah | A permutation-based model for crowd labeling: Optimal estimation and robustness[END_REF] where it is only assumed that the matrix M satisfies some shape-constrained conditions before one (or two) permutations acts on the rows (and possibly on the columns) of M . Quite surprisingly, it has been established in [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF] that, at least in some settings, the matrix M can be estimated at the same rate in those non-parametric models as in classical parametric models by relying on the least-square estimator on the class of permuted bi-isotonic matrices. Unfortunately, the corresponding class of matrices is highly nonconvex and no polynomial-time algorithm is known for computing this least-square estimator. Furthermore, known computationally efficient procedures such as spectral estimators [START_REF] Chatterjee | Matrix estimation by universal singular value thresholding[END_REF][START_REF] Chatterjee | On matrix estimation under monotonicity constraints[END_REF] only achieve sub-optimal convergence rates. This has led several authors to conjecture the existence of computational-statistical trade-offs [START_REF] Flammarion | Optimal rates of statistical seriation[END_REF][START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF]. Despite recent progress in this direction [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF][START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], the fundamental limits of polynomial-time algorithms for this class of problems remain largely unknown.

Arguably, for most applications, the primary objective is to recover the underlying permutation π * acting on the rows or equivalently to rank the experts accordingly. While estimation of the full matrix M is closely related to ranking, it is also of a quite different nature as argued below. In this work, we investigate the estimation of the permutation π * by characterizing the minimax risk for estimating π * in a permuted shape-constrained model, introducing a polynomial-time procedure nearly achieving this risk bound. As a byproduct, we also disprove the existence of a computational-statistical gap for the reconstruction of the matrix M .

Problem formulation

A bounded matrix B ∈ [0, 1] n×d is said to be bi-isotonic if it satisfies B i,j ≤ B i+1,j and B i,j ≤ B i,j+1 for any i ∈ [n -1] and j ∈ [d -1]. Henceforth, we write C BISO for the collection of such n × d bounded bi-isotonic matrices.

In this work, we assume that the matrix M is a row-permuted bi-isotonic matrix as in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF][START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF]. In other words, up to a single permutation π * of [n], the matrix M π * -1 defined by (M π * -1 ) i,j = (M π * -1 (i),j ) is bi-isotonic. From a modeling viewpoint, this amounts to assuming that the d questions are ordered from the most difficult question to the most simple question. The permutation π * is not necessarily unique, but the corresponding permuted matrix M π * -1 is unique. Despite that, we refer, with a slight abuse of terminology, to π * as the oracle permutation. With this definition, π * -1 (i) corresponds to any i-th smallest row (or equivalently expert to use the crowd-sourcing terminology) in the matrix M . In the following, the i th row of M is referred to as expert i, whereas the k th column is referred to as question k.

We consider an observation-scheme where the statistician has partial access to noisy observations Y of M such that

Y = M + E , (1)
where the entries of E are centered, independent, subGaussian -see definition 2.2 of [START_REF] Wainwright | High-dimensional statistics: A non-asymptotic viewpoint[END_REF] -with Orlicz norm at most ζ, but are not necessarily identically distributed. In particular, this model encompasses binary observations Y i,k ∼ Ber(M i,k ) which arise in crowd-labelling problems, in which case we have ζ = 1. In the following, we refer to ζ as the noise level.

As usual in the literature -e.g. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF], we use the Poissonization trick to model the partial observations. Given some λ > 0, which is henceforth referred as the sampling effort, we have N = P oi(λnd) observations of the form

(x t , y t ), t =, 1 . . . , N, (2)
where the position x t is sampled uniformly in [n] × [d], and y t = M xt + E xt is an independent observation of matrix Y of (1) at position x t . Conditionally to N , this scheme is equivalent to a uniform sampling scheme with replacement [START_REF] Mao | Minimax rates and efficient algorithms for noisy sorting[END_REF]. If λ < 1, then a specific entry of Y is sampled at least once with probability 1 -e -λ which is close to λ. More generally, λ corresponds to the expected number of times a specific entry of Y is observed, so that λ > 1 would correspond to the situation where entries are sampled multiple times. Since our aim is to recover the permutation π * from the partial observations (x t , y t ), we consider, for some estimator π, the following error metric

l(π; π * ) = M π-1 -M π * -1 2 F , (3)
where . F stands for the Frobenius norm. This loss quantifies the distance between the matrix M ordered according to the oracle permutation π * and the matrix M ordered according to the estimated permutation. When π * is not unique, the error l(π * , π * ′ ) between any two oracle permutations is zero. If π and π * only differ by a transposition or equivalently if the ranking π and π * only differ on two experts, then l(π; π * ) is twice the square Euclidean distance between the corresponding rows of M . More generally, l(π; π * ) interprets as the sum over all i = 1, . . . , n of the square Euclidean distance between the i-th smallest row of M according to π and according to the oracle ranking π * .

The loss (3) is ubiquitous when one aims at estimating the matrix M in Frobenius norm, that is building an estimator M such that M -M 2 F is as small as possible -see e.g. [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF][START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF]. Indeed, estimating π * by π is a first step towards building an estimator of M by doing as if M π-1 was bi-isotonic. It turns out that the error in M -M 2 F decomposes as the sum of two terms, one of them being l(π, π * ) while the other one does not really depend on π. Conversely, an estimator M can be easily transformed into an estimator π whose loss l(π, π * ) is controlled by M -M 2 F . See [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] for further discussions. In summary, controlling l(π; π * ) is important in order to evaluate to what extent π * is well estimated, but it is also the key stepping stone towards a good estimation of the matrix M .

In some works, the authors directly consider distances on the symmetric group of permutations. Examples include the Kendall tau distance d KT (π, π ′ ) = ∑ (i,j)∶π(i)<π(j) 1{π ′ (i) > π ′ (j)} or the l ∞ distance ππ ′ ∞ = max i∈[n] π(i) -π ′ (i) -see [START_REF] Braverman | Noisy sorting without resampling[END_REF][START_REF] Mao | Minimax rates and efficient algorithms for noisy sorting[END_REF] in the noisy sorting model. However, those distances are not well suited to handle the non-parametric class of bi-isotonic matrices, because to control them we would need to make assumptions on the separation between the rows of the matrix M -see Appendix A of [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF].

Equipped with this notation, we consider the minimax risk of permutation recovery as a function of the number n of experts, the number d of question, the sampling effort λ, and the noise level ζ.

R * [n, d, λ, ζ] = inf π sup π * ∈Πn sup M ∶ M π * -1 ∈C BISO E M π-1 -M π * -1 2 F , (4)
where Π n stands for the collection of all permutations of [n]. In particular, our general aim is to tightly control this minimax risk and, if possible, to provide a computationally efficient procedure achieving this minimax risk. Although our primary interest lies in the estimation of π * , we also consider the minimax estimation risk of M

R * est [n, d, λ, ζ] = inf M sup π * ∈Πn sup M ∶ M π * -1 ∈C BISO E M -M 2 F . (5)
as studied in [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF][START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF][START_REF] Pananjady | Worstcase versus average-case design for estimation from partial pairwise comparisons[END_REF] in order to assess the performances of our computationally efficient procedures.

Related work and open questions

The most relevant body of work to the current paper is that on estimating square matrices M satisfying the so-called strong stochastic transitivity class (SST) [START_REF] Chatterjee | Matrix estimation by universal singular value thresholding[END_REF][START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF]. A matrix M belongs to the SST class if (i) M is skew-symmetric that is M + M T = ee T where e is the constant vector of size n and (ii) there exists a common permutation π * of [n] such that row and column-permuted matrix M π * -1 π * -1 is bi-isotonic. This class is suited for considering pairwise comparisons problems. Shah et al. [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF] consider the full observation setting, namely a setting where each entry of the matrix M is observed once in noise -which is in some sense akin to λ = 1 in our Poissonian scheme 1 . They proved that the minimax risk for estimating M in square Frobenius distance is, up to logarithmic terms, of the order of n and is achieved by the corresponding least-square estimator over the SST class. Unfortunately, this estimator cannot be efficiently computed. They also analyzed an efficient spectral estimator achieving the rate n 32 . This rate is also achieved [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF] by the near-linear time Borda count algorithm CRL that simply ranks the individuals according to the row sums of the observations and then plugs the corresponding permutation to estimate M . See also [START_REF] Chatterjee | Estimation in tournaments and graphs under monotonicity constraints[END_REF] for related results. This led some authors [START_REF] Flammarion | Optimal rates of statistical seriation[END_REF][START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF] to conjecture the existence of a √ n computational gap for SST matrices and for other shape-constrained matrices with unknown permutation.

In crowd-sourcing problems where M ∈ [0, 1] n×d , non-parametric models [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] assume that the matrix M is bi-isotonic up to a permutation π * of the rows (experts) -and sometimes also up to a permutation τ * of the columns (questions) 2 . In this paper as in this literature review, we focus however solely on the case where M is bi-isotonic up to a permutation π * of the rows (experts). Mao et al. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] have established the minimax risk R * est [n, d, λ, 1] for estimating M in the specific case where n ≥ d. In the arguably most interesting regime of partial observations λ ≤ 1, they prove that this minimax risk is of the order of n λ ∧ (nd). This rate is achieved by the inefficient least-square estimator. Furthermore, Mao et al. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] were the first to narrow the conjectured computational gap by introducing a new efficient procedure called one-dimensional sorting. In the square case n = d with full observations, these procedures achieve (up to log terms) the rate n 5 4 for estimating the matrix M , thereby improving over the previous n 3 2 barrier.

Recently, this rate was improved by Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF] in a specific instance of the problem where n = d and one has access to a sub-polynomial number of noisy independent samples of the complete matrix M from (1) -which is akin to our Poissonian scheme for λ being subpolynomial in n, d. They introduce a polynomial-time procedure achieving the rate n 1+o (1) for permutation recovery and matrix estimation which, up to the factor n o (1) , turns out to be minimax optimal for both problems. As a consequence, in this very specific instance, the computational gap turns out to be nonexistent.

There remain important open problems to characterize the estimation of π * and M in crowdsourcing problems.

• Beyond the case n ≥ d handled by Mao et al. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF], the minimax risk R * [n, d, λ, 1] of estimation of the permutation π * -as well as the minimax risk R * est [n, d, λ, 1] of estimation of the matrix M -are unknown. In particular, in the rectangular case where n ≪ d, the number of questions exceeds the number of experts is both relevant from a practical [START_REF] Shah | A permutation-based model for crowd labeling: Optimal estimation and robustness[END_REF] and a conceptual perspective. Indeed, the analysis of the least-square estimator of [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] and related works is based on entropy calculation of the class of permuted bi-isotonic matrices. While the minimax risk R * est [n, d, λ, 1] turns out to be (up to logarithm terms), characterized by this entropy, this is not always the case for the estimation of π * as many matrices M share the same permutation π * . As a consequence, even if we leave aside computational constraints, pinpointing the optimal risk R * [n, d, λ, 1] for estimating π * requires quite different arguments.

• Beyond the toy "over-complete" observation model in the square case n = d of Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], it remains unclear whether there is a computational gap for general rectangular settings with partial observations. .

Our Contributions

Echoing with these open problems, we make the following contributions in this work:

• First, we characterize (up to polylogarithmic multiplicative terms) the minimax risk R * [n, d, λ, ζ] of permutation recovery, this, for all possible number of experts n ≥ 1, number of questions d ≥ 1, noise level ζ ≥ 0, and almost all sampling efforts λ > 0.

When n ≪ d, we prove in particular that R * [n, d, λ, 1] ≪ R * est [n, d, λ, 1] in all nontrivial regimes, highlighting that when n ≪ d, the problem of permutation recovery is statistically easier than the problem of matrix estimation.

• Moreover, we introduce a polynomial-time procedure achieving this risk bound, thereby establishing that there does not exist any significant computational-statistical trade-off for the problem of recovering a single permutation π * . While our procedure borrows some of the ingredients of Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], we need to introduce several new ideas to deal with the significantly more involved case n ≪ d. Since an estimator π of π * can be easily combined with a least-square estimator of a bi-isotonic matrix to estimate the matrix M -see e.g. [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] -we also deduce a polynomial time estimator M which nearly achieves the minimax estimation risk R * est [n, d, λ, 1], thereby proving that this problem does not either exhibit any computational-statistical trade-off, thereby answering the open problem of [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF].

To provide a glimpse of our results, let us describe the minimax risks on the arguably most interesting case where the noise level ζ is of order 1 as in the Bernoulli observation setting and where λ < 1 which corresponds to a partially observed matrix. In Section 4, we establish that the minimax risk R * [n, d, λ, 1] of permutation recovery is (up to polylogarithmic multiplicative terms) of the order of nd 1 6 λ 5 6 ⋀

n 3 4 d 1 4 λ 3 4 + n λ ⋀ nd, (6)
whereas the minimax reconstruction risk R * est [n, d, λ, 1] is of the order

⎡ ⎢ ⎢ ⎢ ⎢ ⎣ nd λ ⋀ nd λ 2 3 (n ∨ d) 2 3 + n λ ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ⋀ nd. (7)
We display in Figure 1 a summary of our results in the specific case where we also have λ = 1 on top of ζ = 1, and will discuss this case more in details, as it highlights one of our main findings.

A first comment is that the minimax risk of matrix estimation R * est [n, d, 1, 1] can be interpreted through the covering numbers of the space of permuted bi-isotonic matrices as in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF].

For n ≥ d both minimax risks -R * [n, d, 1, 1], R * est [n, d, 1, 1] -are of the order of n so that recovering the permutation π * is as hard as estimating the matrix M (up to logarithmic factors). This is the regime studied in the literature, see [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF][START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF]. When the number d of questions is largen ≪ d -then the regimes are more tricky. There are two of them, depending on whether n is larger than d 1 3 or not, and in both regimes R * est [n, d, 1, 1] is significantly larger than R * [n, d, 1, 1]. More regimes appear when we do not restrict ourselves to λ = 1, ζ = 1. This complex picture, as well as the fact that R * [n, d, 1, 1] ≪ R * est [n, d, 1, 1] for n ≪ d -and also in many other configurations of λ, ζ -highlights the fact that the difficulty of estimating π * is not governed by the size of the space of permuted bi-isotonic matrices. As a consequence, even if we leave computational aspects aside, it is not clear that the least-square estimator of [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] achieves optimal risk for estimating the permutation π * and, in any case, entropy-based arguments would lead to suboptimal bounds, at least if we use the same arguments as in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF].

As a byproduct of our results, we also establish the minimax risk -and prove that it is achievable in polynomial time -for another loss function termed l ∞ (π, π * ) (see [START_REF] Vershynin | High-dimensional probability: An introduction with applications in data science[END_REF]) put forward in [START_REF] Chatterjee | Estimation in tournaments and graphs under monotonicity constraints[END_REF][START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] -and we also disprove a conjecture regarding a computational-statistical gap for this loss. See Subsection 4.4.

As our minimax results remain valid in the noiseless case (ζ = 0) where one has access to partial observation of the matrix M itself, we are able to tightly decipher the approximation error which is due to the partial sampling of the matrix M from the stochastic error stemming from noisy observations. In some way, this complements the works of Pananjady et al. [START_REF] Pananjady | Worstcase versus average-case design for estimation from partial pairwise comparisons[END_REF] on the effect of the design in the specific case where the sampled entries are sampled uniformly.

Proof techniques and further comparison with the literature

In order to build a polynomial-time procedure nearly achieving the minimax permutation risk in the partial observation setting (2), we first consider the so-called full observation setting where one has access to poly-logarithmic number Υ of samples Y (0) , Y (1) , . . . , Y (Υ) of the complete n × d matrix. This setting is akin to that of Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF] when they handled the specific square case n = d with noise level ζ = 1.

For this reason, our estimators πHT and πW M introduced in Section 3 share some features with the procedure of [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF]. From a broad perspective, our procedure and theirs build a hierarchical sorting tree using a top-down approach as depicted in Figure 2. We start from the complete set [n] of all experts and build a trisection (O, P, I) of [n], where O (resp. I) contains experts that provably are below (resp. above) the median expert, whereas P contains all the experts for which we cannot certify with high confidence whether they are above or below the median. Then, we recursively trisect the sets O and I as depicted in Figure 2. At the end of the algorithm, we obtain a partial ordering on all the experts which can be used to estimate the oracle permutation π * .

Then, the problem of building a suitable estimator boils down to introducing a suitable trisection procedure. We could naively do this by comparing the row-sums of the observed matrix Y which amounts to comparing the mean ability of each expert, but this is well known to lead to suboptimal performances by a factor √ d -see e.g. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF]. To improve over this rate, we need to compare the experts according to convex combinations of suitable questions. As in [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], we start by selecting suitable blocks of questions by detecting the high-variation regions of the mean empirical expert and combine them with spectral algorithms to select suitable convex combinations of questions. Still, we have to refine significantly their spectral procedure to handle the rectangular case n ≪ d. Equipped with these refinements, which are involved technically, but are built on the ideas developed in [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], we arrive at the estimator πHT (see Section 3) that turns out to be minimax optimal in some regimes of (n, d, ζ).

Unfortunately, this method turns out to be sub-optimal in many regimes, for instance for mild values of n ∈ [d 1 3 , d]. Informally, this is due to the fact that our first estimator πHT as well as that of Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF] build an oblivious hierarchical sorting tree. This means that the trisection method decomposes a group G (0) of experts in the hierarchical sorting tree in (O, P, I) only using the experts in G (0) of the matrix Y . In the related problem of hierarchical clustering, most top-down procedures also share this feature. It turns out that the observations of other experts can help improving the trisection of G (0) . In particular, sets of experts that are close in the ordering -such as G (1) and G (-1) in Figure 2-are sometimes valuable to improve the selection of a suitable convex combination of questions. We emphasize this phenomenon and provide more intuition on it in Section 3, when we introduce a new estimator πW M that builds upon the memory of the sorting tree. This new procedure πW M turns out to be near minimax optimal for all values of (n, d, ζ).

Coming back to the partial observation setting (2), we introduce in Section 4 a reduction scheme which boils down to reducing the number of questions in order to come back to a full observation model for a sub-matrix of size n × d -where d -is possibly much smaller than d. Then, relying on the full observation setting described above, we estimate the permutation π * based on the corresponding reduced matrix. In comparison to the full observation model, we can suffer from an additional bias terms which arises in the reduction process. To handle this, we develop a slight variant πW M -SR of πW M -see Appendix H for details. The resulting procedure turns out to nearly achieve minimax permutation recovery risk for all values (n, d, ζ) and all values of λ. Plugging our procedure to estimating the matrix M , we close all the computational gaps pointed out in Mao et al. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] for the problem of matrix estimation with a single unknown permutation -see Subsection 4.3.

Notation and organization of the manuscript

In the following, c, c 1 , . . . stand for numerical positive constants that may change from line to line. Given a vector u and p ∈ [1, ∞], we write u p for its l p norm. For a matrix A, A F and A op stand for its Frobenius and its operator norm. We write ⌊x⌋ (resp. ⌈x⌉) for the largest (resp. smallest) integer smaller than (resp. larger than) or equal to x.

Although M stands for an n × d matrix, we extend it sometimes in an infinite matrix by setting M i,k = 0 when either i ≤ 0 or k ≤ 0 and M i,k = 1 when either i ≥ n + 1 and k > 0 or k ≥ d + 1 and i > 0. The corresponding infinite matrix M π * -1 which is obtained by permuting the n original rows is still bi-isotonic and takes values in [0, 1]. We shall often work with sub-matrices of M that are restricted to a subset P ⊂ [n] and Q ⊂ [d] of rows and columns, in which case we write that the corresponding matrix M ′ belongs to R P ×Q . More precisely, M ′ is such that, M ′ i,j = M i,j for any i ∈ P and any j ∈ Q. In the following, we write that two sequences or functions u and v satisfy u ≲ v, if there exists a universal constant such that u ≤ cv.

In Section 2, we first consider the complete observation problem, where one has access to a poly-logarithmic number of independent samples of the complete noisy matrix Y . We characterize the minimax risk for permutation recovery and prove that it is achieved by a polynomial-time procedure. In section 3, we describe the corresponding polynomial-time procedure. In Section 4, we deal with the problem of partially observed matrix in the model [START_REF] Ballinger | Decisions, error and heterogeneity[END_REF]. All the proofs are deferred to the appendix.

Analysis of the full observation problem

As explained in the introduction, and following [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], we first consider a slightly different problem where we fully observe a Υ-sample Y = (Y (0) , . . . , Y (Υ-1) ) of the noisy matrix according to the model Y = M +E in [START_REF] Aldous | Exchangeability and related topics, École d'été de probabilités de Saint Flour[END_REF]. Here, Υ should be considered as a polylogarithms in n and d. This is of course not very realistic in applications, but it is simpler to first present our algorithmic procedure in this setting, and it also enables more direct comparison to [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF]. We will explain later in Section 4, how one can transform data in the more realistic partial observation scheme from (2) to this full observation scheme. We will then prove that the algorithm applied to the transformed data is near minimax optimal.

We recall that M is a bi-isotonic matrix, up to an unknown permutation π * of its rows. Besides, the noise matrix E is made of independent mean zero subGaussian entries, with Orlicz norm less than or equal to ζ.

Minimax lower bounds

Before considering ranking procedures, we characterize the minimax risk for the problem of ranking with full observations. For the purpose of the minimax lower bound, we assume that the noise matrix E in ( 1) is made of independent normal random variables with variance ζ 2 . For a permutation π * and a matrix M such that M π * ∈ C BISO , we respectively denote P (π * ,M ) and E (π * ,M ) the corresponding probability and expectations with respect to the Υ independent observations of Y . Define

R F (n, d, ζ) = ζ 2 nd 1 6 ζ 1 3 ∧ n 3 4 d 1 4 ζ 1 2 ∧ n √ d ∧ n 2 3 √ d ζ 1 3 + n . (8)
The following minimax lower bound is stated in a setting where one has access to a polylogarithmic number Υ of full samples to be in line with the analysis of the next subsection. Still, we can forget about the dependency in Υ at first reading.

Theorem 2.1. There exists a universal constant c such that the following holds for any n ≥ 2, d ≥ 1, ζ > 0, and κ > 2. Provided that the sample size Υ is less than or equal to log κ (2nd ζ), we have

inf π sup π * ∈Πn sup M ∶ M π * -1 ∈C BISO E (π * ,M ) M π-1 -M π * -1 2 F ≥ c [log -κ (nd ζ)R F [n, d, ζ] ⋀ nd] . (9)
In fact, this theorem turns out to be a consequence of the minimax lower bound in the partial observation scheme -see Section 4. Together with the risk upper bounds of the next section, (9) characterizes, up to polylogarithmic terms, the minimax risk for estimating π * . The term nd in ( 9) is related to the fact that the loss M π-1 -M π * -1 2 F cannot be larger than nd because the entries of M are in [0, 1].

Mild noise level. The risk bound R F (n, d, ζ) involves five different terms, some of them being significant only when ζ is small in comparison to n and d. As these regimes with very small ζ are arguably quite specific, and to simplify the discussion, we will now detail the minimax lower bound in the specific case when ζ = 1.

R F (n, d) = R F (n, d, 1) = (nd 1 6 ) ⋀(n 3 4 d 1 4 ) + n . (10)
In particular, we recognize three main regimes in [START_REF] Chen | Spectral method and regularized mle are both optimal for top-k ranking[END_REF] that depend on n and d. When the number of experts is relatively small (n ≤ d 1 3 ), the risk is proportional to nd 1 6 . Specifying the result to n = 2, one checks that a square distance d 1 6 is necessary to distinguish two experts. As a consequence, a suitable estimator π should be able to coherently rank experts that are distant by more than d 1 6 in squared Frobenius norm, and then to achieve a risk smaller than nd 1 6 . For larger n ≥ d 1 3 , it is in fact possible to build upon the large number of experts to improve the comparisons between experts using in particular spectral methods. For this reason, the optimal risk is proportional to n for n ≥ d. For an intermediary number of experts n ∈ [d 1 3 , d], the risk is of the order of n

(n, d, ζ) ≍ ζ 2 n √ d when ζ ∈ (0, 1 n∨d ].
In fact, this rate is quite easy to achieve by a polynomial time algorithm in this extreme case. It is proven in various works -see e.g. [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF] that ranking the experts according to the row sum of the matrix correctly compares two experts as long as their square distance is at least ζ 2 √ d (up to logarithmic terms). As a consequence, this simple procedure leads to an error

ζ 2 n √ d. While ζ 2 n

√

d is highly suboptimal in most realistic regimes, it turns out to be tight for extremely low level of noise. Finally, the intermediary rate

R F (n, d, ζ) ≍ ζ 5 3 n 2 3 √
d is achieved for slightly larger values of ζ, but it is less clear how to interpret it.

Minimax upper bounds

In the following, we fix a parameter δ ∈ (0, 1) that will correspond to a small probability. We write ζ -= ζ ∧ 1, where ζ is the noise level. In this section, we analyze two estimators πHT and πW M of π * that are described in Section 3 and more formally defined in Appendix A. The first estimator πHT is based on the construction of an oblivious hierarchical sorting tree. We will later explain all the ingredients of this procedure. In contrast, the second estimator πW M relies on the construction of a hierarchical sorting tree with memory. Both procedures have a computational complexity of the order of log c ( nd ζ-δ )(n 3 + nd 2 ), for some c > 0, which makes them polynomial time -unlike the least square procedure e.g. from [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF].

Theorem 2.2. There exist three numerical constants c, c ′ , and c 0 such that the following holds. Fix δ > 0 and assume that Υ ≥ c 0 log 8 (nd δ). For any permutation π * ∈ Π n and any matrix M such that M π * -1 ∈ C BISO , the oblivious hierarchical sorting tree estimator πHT defined in the next section satisfies

M π-1 HT -M π * -1 2 F ≤ cζ 2 log 10.5 2nd δζ - n 2 3 d 1 3 ζ 2 3 ∧ nd 1 6 ζ 1 3 ∧ n √ d + n ,
with probability at least 1 -c ′ n log 9 ( nd δζ-)δ. If we take δ = ζ 2 (nd) -1 in the above expression, we easily deduce -reminding that the entries of M are in [0, 1] -the following risk bound

E M π-1 HT -M π * -1 2 F ≤ cζ 2 log 10.5 2nd ζ - n 2 3 d 1 3 ζ 2 3 ∧ nd 1 6 ζ 1 3 ∧ n √ d + n .
Comparing this bound with [START_REF] Chen | Spectral method and regularized mle are both optimal for top-k ranking[END_REF] in the specific case where ζ = 1, we observe that πHT achieves the optimal risk nd 1 6 for small n ≤ d 1 3 and the optimal risk n for large n ≥ d. Unfortunately, for mild n ∈ [d 1 3 , d], the risk bound is of the order of n 2 3 d 1 3 , which is significantly higher than the minimax lower bound n 3 4 d 1 4 . To close this gap, we turn to the more refined estimator πW M .

Theorem 2.3. There exist three numerical constants c, c ′ , and c 0 such that the following holds. Fix δ > 0 and assume that Υ ≥ c 0 log 8 (nd (δζ -)). For any permutation π * ∈ Π n and any matrix M such that M π * -1 ∈ C BISO , the hierarchical sorting tree estimator with memory πW M satisfies

M π-1 WM -M π * -1 2 F ≤ c log 11 2nd δζ - R F [n, d, ζ] ⋀ nd , (11)
with probability at least 1 -c ′ n log 9 ( nd δζ-)δ.

As for the previous theorem, this high probability result can be turned into a risk bound by taking δ = ζ 2 (nd). In particular, this risk bound matches, up to polylogarithmic terms, the minimax lower bound [START_REF] Chen | Optimal full ranking from pairwise comparisons[END_REF] for all possible values of n, d, and ζ. As a consequence, the estimator πW M is nearly minimax and this ranking problem does not exhibit any computational gap.

In [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], the polynomial-time estimator πLM of Liu and Moitra achieves the minimax risk in the specific square where n = d and ζ = 1. In all the other regimes, no polynomial-time procedure was previously proved to achieve the minimax risk. In fact, even if we do not restrict our attention to polynomial-time procedures, least-square type procedures studied e.g. in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] provably achieve the minimax risk only in the regime when n ≥ d. As alluded in the introduction -see Equations ( 6) and [START_REF] Chatterjee | On matrix estimation under monotonicity constraints[END_REF], the minimax risks for estimating π * and M differ when n ≤ d, so that achieving the optimal risk for π * is not possible using the classical entropy arguments as in [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF]. This highlights the fact that estimating the permutation π * is significantly more challenging in the regime n ≤ d -both from a statistical and computational perspective -than in the regime n ≥ d handled in [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF].

Consequences for the estimation of the matrix M . Provided that we have estimated π * with Υ-1 independent samples, we could use the last sample Y (Υ) to estimate the matrix M by minimizing the least-square criterion

B = arg min B∈C BISO Y (Υ) πW M -B 2 F and setting M = Bπ -1 W M
. Since the set of bi-isotonic matrices is convex, this estimator is computable efficiently [START_REF] Kyng | Fast, provable algorithms for isotonic regression in all l_p-norms[END_REF]. As argued in Proposition 3.3 of [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] and often used in the ranking literature [START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF][START_REF] Chatterjee | Estimation in tournaments and graphs under monotonicity constraints[END_REF][START_REF] Pananjady | Isotonic regression with unknown permutations: Statistics, computation and adaptation[END_REF], it turns out that, with high probability, the reconstruction error M -M 2 F is (up to polylogarithmic terms) the sum of the expected permutation loss

E M πW M -M π * 2
F and the minimax reconstruction risk of a bi-isotonic matrix inf B sup

B∈C BISO E[ B -B 2
F ] where Y = B + E ′ and E ′ is made of independent subGaussian random variables. Hence, based on πW M and Theorem 2.3, it is easy to construct a polynomial-time estimator of M that is also near minimax-optimal in the sense of Equation ( 5). We will further build upon this remark in Section 4 when we come back to the problem of partial observations of the matrix.

Description of the hierarchical sorting estimators

Let us now describe the construction of the estimators πHT and πW M of π * . The construction is quite long and involves several subroutines. For this reason and to ease the understanding of proof details, we also provide a more formal and longer definition in Appendix A. Afterwards, we comment on the different steps of the procedure and on their connection to the literature in Subsection 3.3.

Define τ

∞ = ⌈4 ⋅ 10 7 log 7 ( nd δ(ζ-) 2 )⌉ and t ∞ = ⌈log(n) log(2)⌉. We define Υ * = 6τ ∞ t ∞
for the total number of independent samples required for the computation of these two estimators.

Hence, we are given independent samples Y = (Y (0) , . . . , Y (Υ * -1) ). From a broad perspective, both procedures are based on the construction of the recursive sorting tree as illustrated in Figure 2. Starting from the root of the tree which corresponds to the set [n] of all experts, we build a partition O, P , I, of [n] in such a way that, with high probability, all the experts in O are below the median expert of [n], all the experts in I are above the median expert of

[n], while the remaining experts in P are those for which we are not able to decipher whether they are below or above the median expert of [n].

Having trisected [n], we recursively trisect the subsets O and I-see Figure 2. Each time, the size of the groups O and I is divided by at least 2. Hence, at depth t ∞ , all the groups of O and I have size at most 1. For each depth t = 0, . . . , t ∞ -1, we use 6τ ∞ new samples. The construction of the tree is described in TreeSort -see Algorithm 1 and is based on the routine BlockSort which performs the trisection of a group into (O, P, I).

Let us now explain how to deduce an estimator π from the final hierarchical sorting tree T . Indeed, the hierarchical sorting tree T induces an order on its leaves as follows. For any groups (O, P, I) sharing the same parent, we say that any descendent of O in the tree T is below P , which, in turn, is below any descendent of I in T . This endows a complete ordering on the leaves of the tree T . Denote G = (G 1 , . . . , G α ) the sequence of leaves of the final tree ranked according to this complete order. For any a ∈ [α], we define the lower bound π - G (G a ) and the upper bound

π + G (G a ) of the ranks of experts in G a by π - G (G a ) ∶= ∑ a ′ <a G a ′ and π + G (G a ) ∶= ∑ a ′ ≤a G a ′ . Finally, we sample π arbitrarily in such a way that π(G a ) = [π - G (a) + 1, π + G (a)] . (12)
In other words, the estimator π ranks the groups G a according to the ordering of the groups endowed by T and, given that, ranks the experts G a uniformly at random. See Appendix A for a more formal definition of the ordering.

Description of the trisection of a leaf G into (O, P, I) with BlockSort

The purpose of BlockSort is to build a trisection of a group G of experts into (O, P, I) where O is made of experts that are, with high probability, below the median expert in G and I is made of experts which are, with high probability, above this median expert. It turns out that this construction is based on τ ∞ iterations of a procedure called DoubleTrisection which is the backbone of our procedure. Intuitively, we shall iteratively detect subgroups of experts that are below (resp. above) the median expert of G which, after τ ∞ iterations, will allow us to obtain O and I.

For technical reasons, our definition is slightly more intricate. We shall simultaneously build two collections (O τ , I τ ) and (O τ , I τ ) of groups, the second one being more conservative. We start with empty sets for (O 0 , I 0 , O 0 , I 0 ) = ∅. Then, at each step τ , we will consider the remaining set of experts G ∖ (O τ ∪ I τ ). Define γ = ⌊ G 2⌋ -O τ for the presumed rank of the median expert of G inside G ∖ (O τ ∪ I τ ). Then, using 6 independent samples, we apply DoubleTrisection(Y, T , G ∖ (O τ ∪ I τ ), γ) to compute four subsets (L τ , U τ ) and (L τ , U τ ). With high probability, it turns out that L τ ⊂ L τ is made of experts below the median expert of G and U τ ⊂ U τ is made of experts above the median expert of G. This allows us to update as follows

O τ +1 = O τ ∪ L τ , I τ +1 = I τ ∪ U τ , O τ +1 = O τ ∪ L τ , I τ +1 = I τ ∪ U τ . ( 13
)
The procedure is summarized in Algorithm 2 below.

Algorithm 1 TreeSort(Y)

Require: 6τ∞t∞ samples Y = (Y (0) , . . . , Y (6τ∞t∞-1) ) Ensure: A tree T and an estimator π 1: Initialize T as the tree with only the root [n] 2: for t = 0, . . . , t∞ -1 do 3:

Take 6τ∞ samples Yt = (Y (6tτ∞) , . . . , Y (6(t+1)τ∞-1) ) 4:

Initialize T ′ = T 5:

for All the leaves G at depth t corresponding to O or I as in Figure 2

do 6: Set (O G , P G , I G ) = BlockSort(Yt, T , G) 7:
Add (O G , P G , I G ) to the tree T ′ 8:

end for 9:

Update T = T ′ 10: end for 11: Set π ∶= π(T ) as in [START_REF] Flammarion | Optimal rates of statistical seriation[END_REF] 12: return T and π

Algorithm 2 BlockSort(Y, T , G)

Require: 6τ∞ samples Y = (Y (0) , . . . , Y (6τ∞-1) ), the tree T , a leaf G in T Ensure: A partition of G into (O, P, I)

1: Set γ = ⌊ G 2⌋ and O0, I0, O0, I0 = ∅ 2: for τ = 0, . . . , τ∞ -1 do 3: Take 6 samples Yτ = (Y (6τ ) , . . . , Y (6τ +5) ) 4: set γ = ⌊ G 2⌋ -Oτ 5: (Lτ , Uτ ), (Lτ , U τ ) = DoubleTrisection(Yτ , T , G ∖ (Oτ ∪ Iτ ), γ) as in Algorithm 3 6: Update Oτ+1 = Oτ ∪ Lτ , Iτ+1 = Iτ ∪ Uτ , Oτ+1 = Oτ ∪ Lτ , Iτ+1 = Iτ ∪ U τ 7: end for 8: return (Oτ ∞ , G ∖ (Oτ ∞ ∪ Iτ ∞ ), Iτ ∞ )

Description of the double trisection procedure

We now describe the trisection procedure DoubleTrisection. For this purpose, we first provide a few definitions.

Definitions

In this subsection, we write Y for one data set sampled according to [START_REF] Aldous | Exchangeability and related topics, École d'été de probabilités de Saint Flour[END_REF]. For the sake of simplicity, we often omit the dependence of Y in the definitions. We write D for the set of all dyadic numbers: D = {2 k ∶ k ∈ Z} and we define the sets R = D ∩ [1, d] and H = D ∩ ζ 2 nd , 1 . The collection R corresponds to the possible scales, that is the number of questions under consideration, whereas the collection H corresponds to the possible heights of variations.

For all r ∈ R, we write Q r = {1, r + 1, 2r + 1, . . . , ⌊ d r ⌋ r + 1} for the regular grid of questions with spacing r. If P ⊂ [n] is a set of experts, we denote y(P ) as the mean of the vectors Y i,⋅ for i ∈ P , that is, for all k ∈ [d], we have y k (P ) = 1 P ∑ i∈P Y i,k . For any r ∈ R, we define Z(Y, P , r) as the aggregation of the matrix Y on blocks of questions of size r and with lines restricted to P . More formally, for any i ∈ P and l ∈ Q r , we have

Z i,l (Y, P , r) = 1 √ r l+r-1 k=l Y i,k and Z i,l (Y, P , r) = 1 √ r l+r-1 k=l y k (P ) . (14)
Both matrices are of size P × Q r . Note that, in the above definition, Z i,l (Y, P , r) and Z i,l (Y, P , r) are rescaled by √ r so that the subGaussian norm remains at most ζ. For any subset Q ⊂ Q r , we also write Z(Y, P , Q, r) for the sub-matrix of Z(Y, P , r) restricted to columns in Q.

Given a matrix Z ∈ R P ×Q , a vector w ∈ R Q + with non-negative components and i, j in P , we say i is (Z, w)-above j (or equivalently that j is (Z, w)-below i) if the projection of Z i,⋅ on the direction w is larger than the projection of Z j,⋅ on w, that is ⟨Z i,⋅ -Z j,⋅ , w⟩ > 0, where ⟨., .⟩ stands for the standard inner product between vectors. Now, for γ ∈ {1, . . . , P }, we can consider the γ-th expert i γ ∈ P such that there are exactly γ -1 experts which are (Z, w)-below i γ . Given a tuning parameter β > 0 to be fixed below, we then define the (Z, w, γ, β)-trisection of P on direction w with respect to pivot index γ and matrix Z as the sets:

⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ U w ∶= U (Z, w, γ, β) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ i ∈ P ∶ ⟨Z i,⋅ -Z iγ ,⋅ , w w 2 ⟩ ≥ β log 2 P δ ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ L w ∶= L(Z, w, γ, β) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ i ∈ P ∶ ⟨Z i,⋅ -Z iγ ,⋅ , w w 2 ⟩ ≤ -β log 2 P δ ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ . (15)
Hence a (Z, w, γ, β)-trisection on direction w and pivot γ consists of two possibly empty disjoint subsets U and L which are respectively taken among the γ -1 experts (resp. the Pγ) which are (Z, w)-above (resp. (Z, w)-below) the expert i γ , with a margin of the order of log( P δ).

Remark that if β < β then U (Z, w, γ, β) ⊂ U (Z, w, γ, β), which means that the trisection of P on direction w becomes more conservative as β increases.

In fact, (15) turns out to be the cornerstone or our procedure. Since the coordinates of w are non-negative, the corresponding row-wise weighted sums of the aggregation E[Z(Y, P , r)] of the signal matrix M are also ordered according to the oracle permutation. In other words, the k-th expert in P has the k-th highest value of the expectation of this weighted sum.

For r ∈ R and Q ⊂ Q r , choosing w = 1 Q in (15) amounts to trisecting P according to the average of the observations over all questions in ⋃ l∈Q [l, l+r). In that case, we write for simplicity [START_REF] Liu | Minimax rates in sparse, high-dimensional changepoint detection[END_REF] simply amounts to ranking experts according to their average over all the questions. As explained in the introduction, the global average does not lead to optimal performances. This is why most following steps in the algorithm amount to selecting suitable blocks Q of questions and directions w.

(L Q , U Q ) = (L 1 Q , U 1 Q ). When Q = Q r and w = 1 Q , then
In the following, the tuning parameters β are set as follows.

β tris = 4 √ 2ζ , β tris = 8 √ 2ζ . (16)

Description of the double trisection procedure

Recall that the purpose of DoubleTrisection is to select subsets (L, U ) and (L, U ) of a group P of experts in such a way that L ⊂ L, U ⊂ U , and experts in L (resp. in U ) are with high probability below (resp. above) the γ-th expert of P .

For that purpose, we have 6 independent samples (Y (s) ) s=1,...,6 sampled from (1) at our disposal. Fix any height h ∈ H and any scale r ∈ R. DoubleTrisection relies on the following steps also described in Algorithm 3.

1. Selection of a suitable subset of questions. Using the first sample Y (1) , we first select a subset Q ⊂ Q r . We postpone the definition of the selection procedure to the next subsection. We will introduce two approaches for this Q ∶= Qcp (P , h, r) as in [START_REF] Pananjady | Worstcase versus average-case design for estimation from partial pairwise comparisons[END_REF] or Q ∶= QW M (T , P , h, r) as in [START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF]. These two definitions respectively correspond to the oblivious estimator πHT and to the estimator with memory πWM . 2. Average-based trisection. Using the second sample Y (2) , we consider the corresponding aggregated matrix Z (2) ∶= Z(Y (2) , P , Q, r) as defined in [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF] which focuses on the selected blocks of questions Q . Then, we consider experts whose corresponding row sums on Z (2) is unusually large or small. More formally, we compute the (Z (2) , 1 Q, γ, β tris )trisection and the (Z (2) , 1 Q, γ, β tris )-trisection of P as defined in [START_REF] Liu | Minimax rates in sparse, high-dimensional changepoint detection[END_REF] and where the tuning parameters β tris and β tris are defined in [START_REF] Luce | Individual choice behavior: A theoretical analysis[END_REF]. This allows us to obtain (L Q, U Q) and (L Q, U Q). 3. PCA-based trisection. Then, we focus on the conservative subset of remaining experts

P = P ∖L Q ∪U Q.
Relying on the samples Y (3) , Y (4) , Y (5) , Y (6) , we build the corresponding aggregated matrices Z (s) ∶= Z(Y (s) , P , Q, r) restricted to the subset P for s = 3, 4, 5.

In principle, we would like to aim at the right singular value of E[Z (3) -Z

] as this would give us a nice direction w on which we could apply [START_REF] Liu | Minimax rates in sparse, high-dimensional changepoint detection[END_REF]. For technical reasons to be explained later, we take a roundabout way, by first computing a vector v indexed by P which, in principle, is not too far from the left singular value of E[Z (3) -Z

]. More precisely, we compute v as follows

v ∶= v( P , Q, r) = arg max v 2 ≤1 v T (Z (3) -Z (3)
) 2 2 - 1 2 v T (Z (3) -Z (3)
-

Z (4) + Z ( 4
)

) 2 2 . ( 17
)
The right-hand side term in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] allows us to deal with the fact that the entries of the noise matrix E in (1) are possibly heteroskedastic. Although there exist more elegant workarounds for heteroskedastic noise (e.g. PCA [START_REF] Zhang | Heteroskedastic pca: Algorithm, optimality, and applications[END_REF]), the analysis in those works does not apply in our non-parametric setting. Moreover, v in (17) corresponds to the leading eigenvector of a square symmetric matrix and can therefore be computed efficiently.

Then, we consider the image ẑ = vT (Z (5) -Z

) ∈ R Q of v. After this, we threshold ẑ and take the absolute values of the components. Thus, we get ŵ+ ∈ R Q defined by ( ŵ+

) l = ẑl 1 ẑl ≥2ζ 2 log(2 Q δ)
for any l ∈ Q. Finally, we consider the last aggregated sample Z (6) ∶= Z(Y (6) , P , Q, r) on the set P ⊃ P of experts. We apply these weights ŵ+ to compute the row-wise weighted sums of Z (6) and discard experts whose corresponding weighted sums is unusually small or large. More formally, we apply (Z (6) , ŵ+ , γ, β)trisection and (Z (6) , ŵ+ , γ, β)-trisection of P as defined in [START_REF] Liu | Minimax rates in sparse, high-dimensional changepoint detection[END_REF]. Doing so we obtain (L ŵ+ , U ŵ+ ) and (L ŵ+ , U ŵ+ ) respectively. In the definition of Z (6) we consider the whole set of experts P instead of the remaining of experts P that have not been discarded because otherwise we should have needed to update the value of γ when applying [START_REF] Liu | Minimax rates in sparse, high-dimensional changepoint detection[END_REF]. Finally, we define the trisections (L, U ) (resp. (L, U )) as the union of the corresponding discarded subsets of experts based on 1 Q and ŵ+ , this for all possible height h ∈ H and scale r ∈ R. We recall that the definition of Q was depending on h and r.

⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ (L, U ) = ⋃ (h,r)∈H×R L Q(h, r) ∪ L ŵ+ (h, r), ⋃ (h,r)∈H×R U Q(h, r) ∪ U ŵ+ (h, r) (L, U ) = ⋃ (h,r)∈H×R L Q(h, r) ∪ L ŵ+ (h, r), ⋃ (h,r)∈H×R U Q(h, r) ∪ U ŵ+ (h, r) . ( 18
)
This whole routine for computing (L, U ), (L, U ) is referred to as DoubleTrisection and is summarized in Algorithm 3. We underline that L ⊂ L ⊂ P and U ⊂ U ⊂ P as we took β tris < β tris .

Algorithm 3 DoubleTrisection((Y (s) ) s=1,...,6 , T , P , γ)

Require: 6 samples (Y (s) )s=1,...,6, a set P , a tree T , a pivot index γ ∈ [1, . . . , P ] Ensure: Two trisections (L, U ) and (L, U ) of P

1: Start from L, U, L, U = ∅ 2: for h ∈ H, r ∈ R do 3:
Compute Q ∶= Qcp(P , h, r) as in [START_REF] Pananjady | Worstcase versus average-case design for estimation from partial pairwise comparisons[END_REF] or Q ∶= QW M (T , P , h, r) as in [START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF] using sample Y (1)

4:

Let Z (s) ∶= Z(Y (s) , P , Q, r), for s ∈ {2, 6} be the aggregated matrices of samples defined as in (14) 5:

Let (L Q, U Q), (L Q, U Q) be resp. the (Z (2) , 1 Q, γ, β) and the (Z (2) , 1 Q, γ, β)-trisections of P as in (15) 6:

Define P = P ∖ (L Q ∪ U Q) and the aggregated samples Z (s) ∶= Z (s) (Y (s) , P , Q, r) for s ∈ {3, 4, 5} 7:

Compute the PCA-like direction v ∶= v( P , Q, r) as in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] 8:

Compute ẑ = vT (Z (5) -Z (5) ) and define ŵ+ by ( ŵ+

) l = ẑl 1 ẑl ≥2ζ 2 log(2 Q δ)
for any l ∈ Q

9:

Let (L ŵ+ , U ŵ+ ), (L ŵ+ , U ŵ+ ) be resp. the (Z (6) , ŵ+ , γ, β) and the (Z (6) , ŵ+ , γ, β)-trisections of P as in (15) 10:

Update

L = L ∪ L ŵ+ ∪ L Q, U = U ∪ U ŵ+ ∪ U Q, L = L ∪ L ŵ+ ∪ L Q, U = U ∪ U ŵ+ ∪ U Q 11: end for 12: return (L, U ), (L, U )
To finish the definition of the estimator, it remains to describe the selection procedures for the suitable blocks of questions that are used in Line 3 of Algorithm 3. As explained above, we consider two procedures Q ∶= Qcp (P , h, r) as in [START_REF] Pananjady | Worstcase versus average-case design for estimation from partial pairwise comparisons[END_REF] or Q ∶= QW M (T , P , h, r) as in [START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF] -which respectively apply to the oblivious estimator πHT and to the estimator πW M with memory.

Definition of Qcp

We start with Qcp (P , h, r). The corresponding estimator πHT is called an oblivious hierarchical sorting tree estimator because Qcp only depends on the restriction of the data to P . As a consequence, the corresponding BlockSort procedure (see Algorithm 2) which builds a trisection of a group G of experts into three subgroups (O, P, I) only depends on the observations on this set G of experts. In other words, the recursive construction of the hierarchical sorting tree estimator is completely oblivious of the rest of the tree. Up to our knowledge, this feature is shared by most hierarchical clustering algorithms.

Fix some height h ∈ H and r ∈ R. Intuitively, Qcp (P , h, r) amounts to focusing on the subset of questions around which the empirical mean expert y(P ) has a high-variation. We provide some intuition on the rationale of this approach in the next subsection. More precisely, we define the CUSUM statistic:

r = 8 32ζ 2 P h 2 log( 2d δ ) ∨ r and Ĉk,r (P ) = 1 r k+r-1 k ′ =k y k ′ (P ) - k-1 k ′ =k-r y k ′ (P ) . (19)
In a nustshell, Ĉk,r (P ) is the empirical variation of y(P ) at question k and at scale r ≥ r.

Then, we define Dcp as the set of questions where the CUSUM statistic is larger than h 4, and Qcp ⊂ Q r for the corresponding subset of blocks or questions of size r.

Dcp = k ∈ [d] ∶ Ĉk,r (y(P )) ≥ h 4 and Qcp = l ∈ Q r ∶ Dcp ∩ [l, l + r) ≠ ∅ . (20)
In [START_REF] Mclaughlin | Stochastic transitivity and cancellation of preferences between bitter-sweet solutions[END_REF], the choice of r ≥ r is due to the fact that we need to compute an empirical mean C k,r (y(P )) on enough questions so that its standard deviation is small compared to h.

Definition of QW M

Finally, we describe QW M (T , P , h, r) which corresponds to the estimator πW M . The set P is a subset of a leaf G of the tree T and we write t for its depth. As illustrated in Figure 2, there is a natural order on the nodes of T at depth t that have been either obtained as subsets of type O or I in BlockSort (Algorithm 2). We can index these nodes according to the ordering by setting G (0) = G and then G (1) , G (2) ,. . . as the following groups. Similarly, G (-1) , G (-2) ,. . . stand for the groups preceding G (0) . See Figure 2 for an illustration. In fact, with high probability, for any a, all the experts in G (a+1) are above the expert in G (a) . As a consequence, the observations in G (1) and G (-1) can bring some informations on the behaviour of the experts in P ⊂ G (0) . Fix r ∈ R and h ∈ H. Define r ∈ R as r = 4(⌈2 9 log(4d R δ) ζ 2 P h 2 ⌉ dya ∨ r), where ⌈x⌉ dya = 2 ⌈log 2 (x)⌉ . As before, r ≥ r stands for the scale which is required if we want to estimate the variation of y(P ) with a standard error small compared to h. Now, we consider any scale r cp ∈ [4r, 2r] ∩ R. The rationale is that, if r cp < r, we can reduce the standard deviations of the empirical means by considering an average over experts in neighboring groups. Define the upper neighborhood V + rcp and lower neighborhood V - rcp as the set of groups above G and below G that are necessary to have enough experts at scale r cp .

a + W M = min{a ∶ G (1) + ⋅ ⋅ ⋅ + G (a) ≥ 2 11 ζ 2 log(4d R δ) rcph 2 } and V + rcp = a + W M ⋃ a=1 G (a) ; (21)
a - W M = min{a ∶ G (-1) + ⋅ ⋅ ⋅ + G (-a) ≥ 2 11 ζ 2 log(4d R δ) rcph 2 } and V - rcp = -1 ⋃ a∈-a - W M G (a) . ( 22
)
For a given subset P ⊂ G, we define the corresponding CUSUM statistic Ĉ(ext) k,rcp computed on the questions [k -r cp , k + r cp ) and using the empirical mean observations in V + rcp ∪ V - rcp if r < 2r and in P if r cp = 2r:

Ĉ(ext) k,rcp = 1 r cp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∑ k+rcp-1 k ′ =k y k ′ (V + rcp ∪ V - rcp ) -∑ k-1 k ′ =k-rcp y k ′ (V + rcp ∪ V - rcp ) if r cp ∈ [8r, 2r) ∑ k+rcp-1 k ′ =k y k ′ (P ) -∑ k-1 k ′ =k-rcp y k ′ (P ) if r cp = 2r (23)
If r cp = 2r, this new definition of the CUSUM with memory matches the definition [START_REF] Mclaughlin | Stochastic transitivity and cancellation of preferences between bitter-sweet solutions[END_REF] in the previous paragraph. For r cp < 2r, we are not able to average on enough expert in P . To deal with this issue, we average on a suitable number of neighboring experts. Beside considering questions around which the variations of y(P ) are large enough, we also check whether, on the corresponding regions, the width of P , that is the difference between the best expert and the worst expert in P is high enough. Given a question k ∈ [d], we define

∆(ext)
k,rcp as the difference between the locals average on [k -r cp , k + r cp ) of the neighbourhoods of G that is

∆(ext) k,rcp = 1 2r cp k+rcp-1 k ′ =k-rcp y k ′ (V + rcp ) -y k ′ (V - rcp ) . (24)
Since the groups G (1) ,. . . , G (2) are above the best expert in P and since the groups G (-1) , G (-2) ,. . . are below the worst expert in P , this statistic ∆(ext) k,rcp overestimates the width of P . In the next subsection, we will explain why it is relevant to consider the width of P .

We are now equipped to define the subsets DW M (T , P , h, r) of suitable questions and the corresponding QW M (T , P , h, r) of corresponding blocks.

DW M = k ∈ [d] ∶ ∃r cp ∈ [4r, r] ∩ R s.t. Ĉ(ext) k,2rcp ≥ h 16 and ∆(ext) k,rcp ≥ h 16 ; (25)
QW M = {l ∈ Q r ∶ DW M ∩ [l, l + r) ≠ ∅} . (26)
In other words, DW M is made of questions for which there exists a scale r cp such that simultaneously the empirical variations Ĉ(ext) k,2rcp at scale 2r cp is at least of order h and the empirical width at scale r cp is at least of order h.

Comments on the procedure and relation to the literature

These twin procedure are quite involved and combine several ingredients, some of them being already used by Liu and Moitra [START_REF] Liu | Minimax rates in sparse, high-dimensional changepoint detection[END_REF]. In particular, they introduced the key ideas of localization of the suitable blocks of questions through change-point detection on the mean expert and of a spectral clustering scheme for dividing blocks of experts. Still, we need to add several key elements in order to deal with the arguably more involved setting where n ≤ d. We describe below how our procedure compares to [START_REF] Liu | Minimax rates in sparse, high-dimensional changepoint detection[END_REF] and highlight also the main differences and new ideas. Also, despite the fact that our procedure is very involved, it remains computationally efficient. Overall, the full procedure requires O[log c ( nd ζ-δ )(nd 2 + n 3 )] operations for some c > 0. Indeed, each of the main steps of the algorithm correspond to matrix multiplications and computations of the largest eigenvector of a square symmetric matrix.

In this subsection, we discuss three key steps of the algorithm: (i) the selection of blocks of questions corresponding to the high-variation regions of the average expert in the group as in the definition of Qcp , (ii) construction of the weights vector w + by a spectral procedure, (iii) the use of neighboring groups in QW M .

Detecting high-variation regions of the average expert

Recall that, for a fixed r ∈ R and h ∈ H, Qcp selects blocks of questions in which the variations [START_REF] Mclaughlin | Stochastic transitivity and cancellation of preferences between bitter-sweet solutions[END_REF] Ĉk,r (P ) of the average y(P ) at question k and at scale r ≥ r is higher than h 4.

To explain the rationale behind this choice, let us first consider a toy example depicted in Figure 3. Assume that the group P is made of two subgroups of experts U * and L * and that all the experts in U * and all the experts in L * are identical. Also, assume that the corresponding rows only differ on r consecutive questions by h and are otherwise identical. As illustrated in Figure 3, it turns out that the expected average expert m(P ) = E[y(P )] needs to vary by h at scale r near the block of questions on which the two groups of experts are differing. This is due to the fact that both the rows corresponding to U * and L * are isotonic and that the row of U * is always larger or equal to that of L * . As a consequence, by restricting our attention to the blocks of questions corresponding to high-variation regions of m(P ) (or in practice y(P )), we are able to much reduce the dimension of the problem and thereby to improve our ability to distinguish different experts.

Beyond this toy example, we show in Lemma D.3 that there exists a suitable scale r ∈ R and a suitable height h ∈ H such that, by restricting our attention to blocks of questions of size r such that the expected average expert m(P ) varies by at least h 2, we are able to retain a significant proportion of the differences between experts in P . In other words, focusing on regions of high-variation of y(P ) in the blocks Qcp is, at least for some scale and some height, a suitable dimension reduction technique. This phenomenon was already observed in [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF] and their procedure also uses such dimension detection techniques. In our paper, we also build upon this idea, which has also important consequences, in a related yet different manner, in the rectangular case where n ≤ d.

If we do not apply the spectral clustering sorting steps in πHT , that is, if we do not compute v and ŵ+ in DoubleTrisection, then we would get a risk bound for πHT of the order of ζ 5 3 nd 1 6 instead of that of Theorem 2.2. In other words, the dimension reduction in Qcp is alone sufficient to recover the optimal risk in the case where d is quite large and ζ is mild -

namely n ≤ ζ 2 3 d 1 3 and ζ ∈ [1 d, √ d].

On the spectral estimation of the weights

In this subsection, we explain how the computation of ẑ in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] and the corresponding weights ŵ+ allow to improve over the ζ 5 3 nd 1 6 rate. Again, we start with a motivating toy example depicted in Figure 4. As previously, we consider a situation where P can be decomposed into two subgroups U * and L * of the same size. The corresponding rows L * are block-constant with blocks of questions of size r and increased by h at the end of each block of questions.

On the other hand, the corresponding lines of U * are, in each block of questions, either equal to the rows of L * , or are exactly at a distance h above. These last blocks of questions are the only ones which are informative when it comes to distinguishing the best experts in the group -namely U * -from the worst experts in the group -namely L * . Some of the blocks corresponding to high-variation regions of the expected average row m(P ) do not convey any information on the difference between U * and L * -see Figure 4. In this example, at scale r, all the blocks of size r are to be detected by the high variation dimension reduction step, that is Qcp = Q r . At the second step, we consider the corresponding aggregated matrix Z -Z at scale r as defined in [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF]. To be more specific, let us assume that L * = U * = 3. Then, Z -Z is a 6 × 8 matrix whose expectation is of the form of the right panel in Figure 4. In this specific example, the rank of this expected matrix is one and some of its columns are completely useless to decipher experts in U * from experts in L * . In contrast, taking w as a right singular vector associated to the largest singular value of this matrix would allow us to select the significant blocks of questions while discarding the irrelevant ones. While this example is very specific, this still sheds some light on why spectral clustering procedure can be of interest for this problem and how it can help recover blocks of questions that are the most informative for dividing the experts.

E[Z -Z] = 1 2 √ rh ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ 0 -1 -1 0 0 -1 0 0 0 1 1 0 0 1 0 0 0 1 1 0 0 1 0 0 0 -1 -1 0 0 -1 0 0 0 -1 -1 0 0 -1 0 0 0 1 1 0 0 1 0 0 ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠
Let us come back to a general matrix M and to the spectral step of DoubleTrisection as described in the previous section. Up to a permutation of its rows, the expectation 3) is isotonic in each column. It turns out that the entries of any left singular vector associated to the largest singular value of Θ (3) -Θ 3) is, up to the permutation, either nonincreasing or non-decreasing. As a consequence, the left-singular value of Θ (3) -Θ 3) can bring information on the underlying ranking. This property is at the heart of spectral ranking algorithms [START_REF] Vigna | Spectral ranking[END_REF]. Unfortunately, contrary to the analysis of spectral methods in the Bradley-Luce-Terry model [START_REF] Chen | Spectral method and regularized mle are both optimal for top-k ranking[END_REF][START_REF] Chen | Optimal full ranking from pairwise comparisons[END_REF], we cannot control the entry-wise deviations of the left singular eigenvector of Z (3) -Z (3) because the matrix Θ (3) -Θ (3) is non-parametric and does not necessarily exhibit any spectral gap. To handle this, Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF] suggest to compute a right singular vector of Z (3) -Z (3) and, using another independent sample, to compare the experts based on the corresponding weighted average of the experts. Unfortunately, while their analysis provides near optimal results for n = d, this would not work for n ≤ d. In DoubleTrisection, we apply a more involved workaround (i) to handle possible heteroskedastic noise and (ii) to improve the convergence rates in comparison to Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF]. Indeed, we first compute in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] a debiased version v of the left-singular vector of Z (3) -Z (3) . Then, we compute the image

Θ (3) -Θ (3) of Z (3) -Z (
[Z (5) -Z (5)
] T v, threshold it, and take its absolute value to obtain our estimated weights ŵ+ . In principle, ŵ+ aims at being close to the right first singular vector of Θ (3) -Θ (3) . In comparison to Liu and Moitra [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], ŵ+ better handles the situation where the matrix 3) is highly rectangular (with many columns) and where its corresponding right singular vector is nearly sparse.

Θ (3) -Θ (

On the tree information and the definition of QW M

The oblivious estimator πHT based on Qcp is only proved to achieve the suboptimal error of Theorem 2.2. In this section, we explain how QW M improves the performances of the procedure by relying on the neighboring experts to fix one possible weakness of Qcp and so, improve the dimension reduction step.

Indeed, Qcp selects spurious blocks of questions. In the previous toy example (Figure 4), some of the blocks corresponding to high-variation values of the expected mean expert m(P ) do not bring any suitable information for ordering the experts in P because, in these blocks, all the experts are close to each other. In other words, the width of P , that is the difference between the best and worst experts in P , is small. It is not possible to easily estimate this width from the observations in P since this would require to have sorted the experts in P in the first place. Still, we can estimate this width by comparing the average of experts that are above P with average of experts that are below P . A first idea would therefore be to consider a large enough number of experts above and below P in order to estimate the width with a small variance and to exclude regions such that the estimated width on a window of size r is small compared to h. This is exactly the purpose of the statistic ∆(ext) k,r . The selected blocks QW M only contain regions such that the estimated width

∆(ext) k,r
is large enough compared to h -see the left panel in Figure 5. Unfortunately, the statistic

∆(ext) k,r
may suffer from a large positive bias if the experts above or below P are away from P . Moreover, considering only the scale r is not sufficient because we are forced to average over many experts above and below P in order to have a small variance at this small scale, leading to a large bias. For this reason, we consider all possible scales r cp (in a dyadic grid) between r and r.

Another important idea in the dimension reduction scheme is the following: If there is a region of questions in which, not only the mean experts of the group P but also the mean experts in neighboring groups of P have a high variation, it is interesting to detect this highvariation region by relying on all these neighboring groups in order to decrease the variance of the CUSUM statistic. With this idea, we are able to consider the CUSUM statistic at a smaller scale r cp ≤ r -see the right panel in Figure 5. This is exactly the purpose of the statistic

Ĉ(ext) k,rcp .
In our procedure, we build a collection DW M that selects a question k if there exists a scale r cp in [4r, r] such that both the CUSUM statistic Ĉ(ext) k,2rcp at scale 2r cp is large and the empirical width ∆(ext) k,rcp is large. This combines the two ideas described in the previous paragraphs which, in turn, allows us to further reduce the dimension in comparison to Dcp while ensuring that the selected questions in DW M contains all the relevant regions to trisect P , namely regions of size r, on which P has a variation at least of the order of h and the width of P is at least of the order of h.

Interestingly, in the square case where n = d considered in [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF] or more generally when n ≥ d, this dimension reduction variant is not necessary to achieve the minimax risk as the oblivious estimator πHT is already optimal. The dimension reduction scheme QW M allows us to improve the risk bound from that Theorem 2.2 to that of Theorem 2.3. In the specific case where the noise level ζ is equal to one, the term n 2 3 d 1 3 in the risk bound is improved to the optimal one n 3 4 d 1 4 . Hence, building upon the neighboring groups in QW M turns out to be the key ingredient to recover the minimax risk in the large d regime where n ∈ [d 1 3 , d]. ) and m(V - rcp ) respectively correspond to the expected average experts in the neighboring groups V + rcp and V - rcp defined in [START_REF] Pananjady | Isotonic regression with unknown permutations: Statistics, computation and adaptation[END_REF]. In the left panel, the third and fourth blocks are not selected because the corresponding statistic ∆(ext) k,rcp is small. In the right panel, both the statistic Ĉ(ext) k,rcp and ∆(ext) k,rcp are large compared to h. If P is small, this block is selected using a scale r cp ≲ r.

Partial observations

We now come back to the partial observation setting. Given λ > 0, we are given P oi(λnd) independent observations (x t , y t ) where x t is sampled uniformly in [n] × [d] and, conditionally to x t , y t = M xt + E xt is an observation of the full model (1) at position x t . As noted above, λ stands for the sampling effort and the larger λ, the more samples on average.

Minimax Lower bound

As in Section 2.1, we first state a minimax lower bound in the case where the noise matrix E is made of independent Gaussian random variables with variance ζ 2 . Note that the following minimax lower bound also handle the noise case where ζ = 0, i.e. the noiseless case.

M ∶ M π * -1 ∈C BISO E (π * ,M ) [ M π-1 -M π * -1 2 F ] ≥ c R F [n, d, ζ √ λ] + n λ e -2λ ⋀ nd . ( 27
)
As in the previous minimax lower bound, the quantity nd simply appears because the entries of M lie in [0, 1]. In ( 27 does not depend on ζ and is also present in the noiseless setting. It simply quantifies the fact that, for λ < 1, observations are lacking so that it is impossible to correctly rank experts if there are no observations on the questions on which they are distinct. As the minimax lower bound in ( 27) turns out to be nearly tight in light of Theorem 4.3, we refer to [START_REF] Shah | A permutation-based model for crowd labeling: Optimal estimation and robustness[END_REF] as the minimax risk in the following. For the purpose of the discussion, we will first focus on the case where ζ = 1 and λ < 1, which corresponds to the case where we really have partial observations on the matrix. We will then turn to ζ = 1 and λ > 1, which corresponds to the case where we observe several times each entry of the matrix. Finally, we discuss the noiseless case where ζ = 0.

Low-sample size. We first focus on the case where ζ = 1 and λ < 1, which corresponds to the case where we really have partial observations on the matrix. If λ ≤ 1 d, then the minimax risk is of the order of nd and it is impossible to perform significantly better than a random guess. This is not surprising as there are, in expectation, less than one observation on each row. For λ ∈ [1 d, 1], the minimax risk is of the order of

nd 1 6 λ 5 6 ⋀ n 3 4 d 1 4 λ 3 4 + n λ
In the rectangular case where n ≥ d, the minimax risk is then of the order of n λ

for λ ∈ [1 d, 1]. When n ∈ [d 1 3 , d], the minimax risk is of the order of n λ for λ ∈ [1 d, n d],
and of the order of n 3 4 d 1 4

λ 3 4 for λ ∈ [n d, 1].
For even smaller n ≤ d 1 3 , there is one more regime since

R F [n, d, 1 √ λ] ≍ ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ n λ if λ ∈ [ 1 d , n d ] ; n 3 4 d 1 4 λ 3 4 if λ ∈ [ n d , n 3 d ] ; nd 1 6 λ 5 6 if λ ∈ [ n 3 d , 1] .
Large-sample size. In the setting where λ > 1 and ζ = 1, there are several observations per entries. In this case, there are many regimes in ( 27) that depend on n, d, ζ, and λ. To simplify the discussion, we focus here on the case n = d and ζ = 1. Then, the minimax risk is of the order of n 3 4 d 1 4

λ 3 4 for λ ∈ [1, n 2 ]
and is of the order of n √ d λ for λ ≥ n 2 . This 'easy rate' n √ d λ is achieved by the simple procedure that ranks the experts according to the row sums [START_REF] Shah | Stochastically transitive models for pairwise comparisons: Statistical and computational issues[END_REF][START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF]. This simple method turns out to be optimal in the regime where there are more than n 2 observations per entry.

Noiseless case. In the extreme case where ζ = 0 and λ ≥ 1 d, the minimax risk is of the order of (n λ) × e -2λ , which, for some small λ is of the order of n λ. This minimax lower bound is quite simple to prove. Without loss of generality, suppose that 1 λ is an integer. Consider a matrix M such that all its columns, except its 1 λ first ones are constant and equal to one, so that it boils down to considering a ranking problem of size n × (1 λ). In this reduced model, there are two types of experts: (a) experts that are constant and equal to zero and (b) experts that are constant and equal to one. Obviously, if one is given at least one noiseless observation on a row, then it is possible to assign it to a group. However, on each row there is a probability e -1 of having no observations. Hence, on expectations there are n e experts that are impossible to classify. For this reason, any estimator must suffer from a risk at least of the order n λ.

Reduction to the full observation model

We now describe a scheme to adapt the estimators πHT and πW M that we developed in the full observation setting of Section 2, to this more general Poissonian setting (2), which encompasses the partial observation setting as well as the over-complete observation setting where each entry is sampled several times. Roughly, if λ is small, we simply decrease the number of columns of the matrix M in order to obtain a reduced matrix with full observations. Conversely, if λ is really large, which corresponds to the case of multiple observations per entry, we simply average the multiple observations per entry to reduce the noise levels.

As in Section 2, we fix δ ∈ (0, 1) that will correspond to a small probability. Given this δ > 0, we denote

Υ * = Υ * (n, d, ζ √ λ ∨ 1)
the number of independent samples required in Section 2 for the estimation through πHT or πW M of the n × d matrix M with a noise level equal to

ζ √ λ ∨ 1. Recall that Υ * is of the order of log 8 (nd(λ ∨ 1) (δζ -)). Define λ -= λ [4Υ * ].
For any i ∈ [n] and any S ⊂ [d], we write n i,S the number of observations in the sample falling in {i} × S, that is n i,S = {t ∶ x t ∈ {i} × S} . The following lemma is a simple consequence of Chernoff inequality for Poisson random variables. Lemma 4.2. Assume that λ -∈ [2 d, 1], we fix l(λ) = ⌊1 λ -⌋. With probability higher than 1 -δ, we have min

i∈[n] min j∈[⌊d l(λ)⌋] n i,[(j-1)l(λ)+1,jl(λ)] ≥ Υ * .
Now assume that λ -> 1. With probability higher than 1 -δ, we have

min i∈[n] min j∈[d] n i,{j} ≥ λ -Υ * .
Henceforth, we work under the event introduced in the previous lemma. If this event does not hold, we choose πW M P arbitrarily. To build πW M P , we consider three subcases that depend on the value of λ:

1. Very small sample size. If λ -≤ 2 d, then we simply choose πW M P uniformly at random over the set of all possible permutations. While this choice does not depend on the data and could therefore seem sub-optimal, it is not the case, as the minimax lower bound states that it is impossible to perform better than random guess in this setting.

2. Small sample size. If λ -∈ [2 d, 1], then we build Υ * matrices Y ↓ = (Y ↓(0) , Y ↓(1) , Y ↓(Υ * -1) ) of size n × ⌊d l(λ)⌋ in the following way. For any i ∈ [n], j ∈ [⌊d l(λ)⌋] and s ∈ [0, Υ * -1], Y ↓(s) i,j = y t where t is the (s + 1)-th observation such that x t ∈ {i} × [l(λ)j + 1, l(λ)(j + 1)
]. On the event of Lemma 4.2, this definition is valid as we observe enough samples for any i, j. Then, we compute πW M P as the variant πW M -SR , introduced in Section H, applied to this sample of reduced matrices.

Large sample size. If λ

-≥ 1, then we build Υ * matrices Y ↓ = (Y ↓(0) , Y ↓(1) , Y ↓(Υ * -1) )
of size n × d in the following way. For any i ∈

[n], j ∈ [d], l ∈ [⌊λ -⌋], and s ∈ [0, Υ * -1], define Y ↓(s) i,j
= 1 ⌊λ -⌋ ∑ t y t where the y t 's are the z-th observations such that x t = (i, j) with z ∈ [1 + (s -1)⌊λ -⌋, s⌊λ -⌋]. In other words, we build the samples Y ↓ be averaging ⌊λ -⌋ observations on each entries. Again, on the event of Lemma 4.2, this definition is valid as we observed enough samples for any i, j. Then, we define πW M P as πW M applied to this sample of averaged matrices. By averaging the independent observations, we reduce the noise level of each entry from

ζ to ζ √ λ -.
For λ -≤ 2 d, there are very few observations on each row so that it is very difficult to compare the experts. For λ -∈ [2 d, 1], we have access to less than Υ * noisy observations of the matrix M . The rationale of our procedure is to group together l(λ) consecutive questions together in such a way that there are enough observations on each of these groups. The resulting matrices of observations Y ↓(s) have around λd Υ * columns. We could have applied the procedure πW M defined in the previous section to Y ↓ , but the corresponding subGaussian norm of the noise would be 1 + ζ (instead of ζ) because there is additional variability coming from the fact that any entry in the reduced matrices has been sampled uniformly among l(λ) entries in the original matrices. This would lead us to a procedure achieving the minimax rate with respect to n, d, and λ but with a suboptimal dependency with respect to ζ since ζ would be replaced by ζ + 1. This is the reason why, for λ -∈ [2 d, 1], we rely on a slight variant πW M -SR (see Section H) of πW M that builds upon the fact that the variations that are due to the aggregation of M are very specific. Theorem 4.3. There exist four numerical constants c 1 -c 4 such that the following holds. Fix

δ = ζ 2 -[(λ ∨ 1)nd] -2 .
For any permutation π * ∈ Π n and any matrix M such that M π * -1 ∈ C BISO , the sorting tree estimator πW M P defined above satisfies

E M π-1 W M P -M π * -1 2 F ≤ c 1 log c 2 nd(λ∨1) ζ- R F (n, d, ζλ -1 2 ) + n λ e -c 3 λ log -c 4 ( nd(λ∨1) ζ- ) . (28)
Up to logarithmic terms and up to the logarithmic term inside the exponential term (28), both the minimax upper bound [START_REF] Thurstone | A law of comparative judgment[END_REF] and lower bound [START_REF] Shah | A permutation-based model for crowd labeling: Optimal estimation and robustness[END_REF] match for all values of n, d, λ and ζ. As a consequence, this problem of estimating a single permutation π * does not exhibit any significant computational gap.

Let us further discuss and compare the exponential term nλ -1 e

-c 3 λ log -c 4 ( nd(λ∨1) ζ-
) in ( 27) and nλ -1 e -2λ in [START_REF] Thurstone | A law of comparative judgment[END_REF]. First, observe that these two terms are larger than R F (n, d, ζλ -1 2 ) only when the noise level ζ is small, so that it is relevant to discuss them only when ζ ≪ 1. Second, note that there is a significant mismatch between these exponential terms only when λ is close to one, up to a polylogarithmic factor, since otherwise, either the exponential is close to one (for λ ≤ 1) or the exponential is so small that it becomes negligible in comparison to R F (n, d, ζλ -1 2 ). One may object that the logarithmic term log -c 4 (

nd(λ∨1) ζ-
) may be large in case ζ is really small -think e.g. of ζ = e -nd . Let us consider this extremely low noise setting where, say ζ ≤ 1 (nd) 2 . If one applies the procedure πW M P with ζ 0 = 1 (nd) 2 ≥ ζ, then the logarithmic terms become bounded inside the exponential. Since R F (n, d, ζ 0 λ -1 2 ) is always smaller than nd ⋀ n λ e -c 3 λ provided that λ ≤ 1, this estimator achieves the risk bound n λ ∧ nd, which is optimal for all ζ ∈ [0, 1 (nd) 2 ] and all λ ≤ 1. To sum up, there is gap between our minimax lower and upper bounds only either (i) in the low-noise level with large but mild sampling effort, that is

ζ = o(log -c (nd)), ζ ≥ (nd) -2 , and λ ∈ [log c log(nd), log c ′ (nd)]
for some c and c ′ > 0 or (ii) in the extremely low noise level with large sampling effort, that is ζ ≤ (nd) -2 and λ ≥ 1.

In πW M P , we have plugged in the hierarchical sorting tree estimator with memory πW M . If we had plugged in the oblivious hierarchical sorting tree estimator πHT , then the resulting estimator would satisfy a similar rate similar to [START_REF] Thurstone | A law of comparative judgment[END_REF] except that the term n 3 4 d 1 4 λ 3 4 would be replaced by the slower rate n 2 3 d 1 3 λ 2 3 .

Reconstruction of the matrix M

In this subsection, we assume again that the noise level ζ = 1 to simplify the exposition. As alluded in Section 2, it is quite straightforward to estimate the matrix M and control the corresponding loss M -M 2 F by a simple subsampling step explained e.g. in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] that we recall here. First, we split the sample into two part by assigning independently each observation to the first subsample with probability 1 2 and the second subsample with probability 1 2. Then, we use the first subsample to estimate the permutation π of the experts. As for the second subsample (x

(2) t , y (2)
t ), we define the empirical observed matrix Y (2) by

Y (2) i,j = 1 λ t y (2) t 1 x (2) t =(i,j) .
Then, we compute the least-square estimator M π of M π in the class of bi-isotonic matrix

M π = arg min B∈C BISO B -Y (2) π 2
F . This estimator can be computed in near linear-time [START_REF] Kyng | Fast, provable algorithms for isotonic regression in all l_p-norms[END_REF]. Then, Proposition 3.3 in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] states, that with high probability, the loss M -M 2 F is, up to logarithmic terms, smaller than the sum of the minimax risk for estimating a bi-isotonic matrix B and the loss M π-1 -M π * -1 2 F . Plugging this proposition with our estimator πW M P with δ = (λ ∨ 1) (np), we readily arrive to the following risk bound for the corresponding estimator

M W M P . Define R 1 (n, d, λ) = nd λ ⋀ nd λ 2 3 (n∨d) 2 3 ⋀ nd λ .
Mao et al. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] have proved that, up to polylogarithmic factor and up to a possible additive term (n ∧ d) λ, the minimax risk in square Frobenius norm for estimating a bi-isotonic matrix with partial observations is R 1 (n, d, λ). Corollary 4.4. There exist two numerical constants c and c ′ such that the following holds. For any permutation π * ∈ Π n and any matrix M such that M π * ∈ C BISO , we have

E M W M P -M 2 F ≤ (nd) ⋀ c log c ′ ((λ ∨ 1)nd) R 1 (n, d, λ) + R F (n, d, λ -1 2 ) ≤ (nd) ⋀ c log c ′ ((λ ∨ 1)nd) R 1 (n, d, λ) + n λ . ( 29
)
The proof is a straightforward consequence of Proposition 3.3 in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] and Theorem 4.3 and is therefore omitted. It turns out that R F (n, d, λ -1 2 ) is always smaller than R 1 (n, d, λ) + n λ , so that the cost of reconstruction for not knowing π * is n λ.

This risk bound ( 29) is minimax optimal, up to polylogarithms, and this for all possible values of n ≥ 2, d, and λ > 0. Indeed, in their Theorem 3.1, Mao et al. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] provide a matching minimax lower bound in R 1 (n, d, λ) in the specific case where n ≥ d, but their proof easily extends to the case where n ≤ d. Besides, our proof of the minimax lower bound n λ in Theorem 4.1 for the problem of estimating π * straightforwardly extends to the problem of matrix estimation (recall that we consider ζ = 1 here).

The least-square estimator πLS of Mao et al. has also been proved to achieve the minimax risk for n ≥ d -see their theorem 3.1 in [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF]. However, no efficient algorithm is known for computing this estimator in πLS , so that our estimator M W M P is, to the best of our knowledge, the first efficient minimax-optimal estimator for estimating M in this context, for any values of n, d, λ.

Bounds for the max loss of Mao et al. [17]

In [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF], Mao et al. control, for an estimator π of the permutation, a different loss from ours. Up to normalization factors, they indeed focus on the maximum l 2 norm of the rows of (M π-1 ) i,.

-

(M π * -1 ) i,. , that is l ∞ (π, π * ) = sup i∈[n] (M π-1 ) i,. -(M π * -1 ) i,. 2 2 .
(

) 30
This loss also considered in [START_REF] Shah | Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons[END_REF][START_REF] Chatterjee | Estimation in tournaments and graphs under monotonicity constraints[END_REF] corresponds to some maximum error of the estimated permutation so that l

∞ (π, π * ) ≥ M π-1 -M π * -1 2 F n. Alternatively, we can define the loss l err l err (π, π * ) = max i,j∈[n] ∶ π(i)<π(j) and π * (i)>π * (j) M i,. -M j,. 2 2 ,
which quantifies the maximum distance between two experts that have not been ranked in a consistent manner. The loss l ∞ and l err turn out to be equivalent as stated in the following lemma.

Lemma 4.5. For any permutation π, we have

l ∞ (π, π * ) ≤ l err (π, π * ) ≤ 4l ∞ (π, π * ) (31)
To simplify the discussion in this section, we assume again that the noise level ζ equals one. Mao et al. [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] provide a simple polynomial time πref achieving

E[l ∞ (π ref , π * )] ≲ d ⋀ d 1 4 λ 3 4 log 3 4 (n) . (32)
Conversely, they prove in their Theorem 3.7 that any estimator π that only ranks the experts i and j according to the differences of the observations on the rows i and j must incur this risk bound-see [START_REF] Mao | Towards optimal estimation of bivariate isotonic matrices with unknown permutations[END_REF] for further details. Besides, they conjecture that the risk bound (32) cannot be improved. In [START_REF] Liu | Better algorithms for estimating non-parametric models in crowdsourcing and rank aggregation[END_REF], Liu and Moitra already pointed out that the max loss l ∞ (π, π * ) is less suited than the loss M π -M π * 2 F for the purpose of estimating the matrix M -see the discussion in the previous subsection. Still, controlling the max loss l ∞ (π, π * ) may be an objective per se, and the study of its minimax value and of the existence of related minimax estimators is relevant. In the following proposition, which is mainly a consequence of our results and proof techniques, we disprove Mao et al.'s conjecture by introducing an estimator πP C achieving a faster rate than [START_REF] Wainwright | High-dimensional statistics: A non-asymptotic viewpoint[END_REF]. Besides, this rate turns out to be minimax-optimal. Proposition 4.6. There exist numerical constants c, c ′ , and c ′′ such that the following result holds. There exists a polynomial-time estimator πP C that performs pair-wise comparisons between the experts and that achieves the risk bound

E[l ∞ (π P C , π * )] ≤ c log c ′ (nd(λ ∨ 1)) d 1 6 λ 5 6 ⋀ √ d λ ⋀ d . (33)
Conversely, for any n ≥ 2, any d ≥ 1, and λ > 0, we have

inf π sup π * ∈Πn sup M ∶ M π * -1 ∈C BISO E (π * ,M ) [l ∞ (π, π * )] ≥ c ′′ d 1 6 λ 5 6 ⋀ √ d λ ⋀ d . (34)
For λ ≤ 1 d, it is not possible to perform significantly better than random guess. Then, in the interesting regime λ ∈ [1 d, d 2 ], the risk is of the order of d 1 6 λ 5 6 . It turns out that this rate corresponds, up to polylogarithmic terms, to the minimal distance between two experts so that one is able to consistently compare them. For very large sample size λ ≥ d 2 , we arrive at the easy regime which is of the order of

√ d λ .
The estimator πP C is based on pairwise comparisons. For any two experts i and j, we apply the procedure πW M P to i and j with δ = [(λ ∨ 1)(n 2 d)] -2 . If the trisection (O, P, I) is of the form (∅, {i}, {j}), we return i ≺ j. If the trisection (O, P, I) is of the form (∅, {j}, {i}), we return j ≺ i. Otherwise, we return nothing. Applying this comparison algorithm to all (i, j), we recover a set of pairwise comparisons PC = {(i, j) ∶ i ≺ j}. With high probability -see the proof for more details-, it turns that PC satisfies two properties:

(i) PC is consistent. For any (i, j) ∈ PC, we have π * (i) < π * (j). (ii) PC contains all 2-tuple of experts that are far apart. More precisely, PC contains all (i, j) such that π * (i) < π * (j), and

M i,. -M j,. 2 2 ≥ c log c ′ (nd(λ ∨ 1)) d 1 6 λ 5 6 ⋀ √ d λ ⋀ d , (35)
for suitable constants c and c ′ .

Then, define the function φ ∶ [n] ↦ N by φ(j) = {(i, j) ∶ (i, j) ∈ PC} which simply counts the number of experts i that are detected to be lower than j. Finally, we build πP C as any permutation that ranks the experts consistently with φ.

In fact, the procedure for computing πP C could be greatly simplified. Indeed, as we only perform pairwise comparisons, some parts of BlockSort turn out to be irrelevant. For instance, the PCA steps are not required. Besides, the sample splits could be avoided and it could even be possible to work with a single observation. As the problem of optimal permutation recovery with respect to the l ∞ loss is not the main scope of this paper, we do not provide a simplified and dedicated algorithm. Besides, we conjecture that our original estimator πW M P also achieves the minimax risk [START_REF] Wu | Lecture notes on information-theoretic methods for high-dimensional statistics[END_REF] with respect to the l ∞ loss.

Appendix A: Full description of the procedures

In this section, we provide a fuller description of the estimators πHT and πW M as a collection of algorithms. We will rely on this description in the analysis of these estimators. To ease its understanding, we make this section completely self-contained. As a consequence, the material presented here is partly redundant with Section 3.

A.1. Sorting a group of experts

Some of the notation have already been introduced in Section 3. Still we define them again here for the sake of completeness. We write D for the set of all dyadic numbers, that is

D = {2 k ∶ k ∈ Z}. Equipped with D, let R = D ∩ [1, d] and H = D ∩ ζ 2 nd , 1 , (36)
respectively denote the dyadic collection of numbers beween 1 and d and the dyadic collection of numbers between 1 nd and 1.

Besides for an integer r ∈ R, we write Q r for the regular grid of [d] of width r:

Q r = 1, r + 1, 2r + 1, . . . ⌊ d r ⌋ r + 1 .
In contrast to Section 3, we start by describing the simple comparison routine before moving to the dimension reduction techniques and to the general architecture of the procedures.

Given a collection P of experts, some data Z ∈ R P ×Q and a direction w ∈ (R + ) Q and a pivot γ ∈ [1 ∶ P ], the following pivoting algorithm sorts the experts in P according to the projection of the data onto the vector w. More precisely, it returns four subsets L ⊂ L and U ⊂ U of experts such that the γ-th best expert according to the (Z, w)-order -as defined above Equation ( 15) -is significantly above all experts in L and below all experts in U . The subsets L and L (resp. U and U ) differ in the level of significance we require. We define the tuning parameters β tris and β tris for Pivot

β tris = 4 √ 2ζ , β tris = 8 √ 2ζ. ( 37
)
Algorithm 4 Pivot(Z, w, γ)

Require: A matrix Z ∈ R P ×Q with P ⊂ [n] a set of experts and Q ⊂ [d] a set of blocks, a direction w ∈ R Q + , w ≠ 0 and a pivot index γ Ensure: Two couples of subsets (L, U ) and (L, U ) of P 1: for i ∈ P do 2:

Compute the statistic ψ(i, w) = ⟨Zi,⋅, w w 2 ⟩ 3: end for 4: Sort the statistics ψ(i, w) : ψ(i1, w) ≤ ⋅ ⋅ ⋅ ≤ ψ(i P , w)

5: U = {i ∈ P ∶ ψ(i, w) > ψ(iγ, w) + βtris log 2 P δ } 6: U = {i ∈ P ∶ ψ(i, w) > ψ(iγ, w) + β tris log 2 P δ } 7: L = {i ∈ P ∶ ψ(i, w) < ψ(iγ, w) -βtris log 2 P δ } 8: L = {i ∈ P ∶ ψ(i, w) < ψ(iγ, w) -β tris log 2 P δ } 9: return (L, U ), (L, U )
When the vector w is equal to 1 Q , we simply write Pivot(Z, γ) instead of Pivot(Z, 1 Q , γ) for the sake of simplicity.

Pivot(Z, γ) = Pivot(Z, w = 1 Q , γ) . ( 38
)
In fact, Pivot(Z, γ) simply amounts to comparing the row sums of Z for each of the experts in P .

In the next two pages, we redefine in more detail the Double Trisection algorithm of Section 3. First, DoubleTrisection -PCA relies on a PCA-type argument to find a suitable direction ŵ+ and then provides two trisections of the subset P of experts using the Pivot sub-routine.

Algorithm 5 DoubleTrisection -PCA(Z, γ)

Require: 4 reduced samples Z = (Z (1) , Z (2) , Z (3) , Z (4) ) where Z (1) , Z (2) , Z (3) ∈ R P ×Q and Z (4) ∈ R P ×Q with some P ⊂ P , and a pivot index γ Ensure: Four subsets (Lpca, Upca) and (Lpca, U pca) of P 1: Compute the following vector with coefficients in P :

v = arg max v ≤1 v T (Z (1) -Z (1) ) 2 2 - 1 2 v T (Z (1) -Z (1) -Z (2) + Z (2) ) 2 2 ∈ R P 2: ẑ = v T Z (3) ∈ R Q 3: Define ŵ+ by ( ŵ+ ) l = ẑl 1 ẑl ≥2ζ 2 log(2 Q δ)
4: (Lpca, Upca), (Lpca, U pca) = Pivot(Z (4) , ŵ+ , γ) 5: return (Lpca, Upca), (Lpca, U pca)

Next, DoubleTrisection -Local(Z, γ) builds two trisections of P based on the reduced samples. First, it builds these trisections by simply using the row sums on the data and then it improves them thanks to DoubleTrisection -PCA.

Algorithm 6 DoubleTrisection -Local(Z, γ) Require: 5 reduced samples Z = (Z (1) , Z (2) , Z (3) , Z (4) , Z (5) ) in R P ×Q , a pivot index γ and a threshold βtris Ensure: Two couples of subsets (L, U ) and (L, U ) of P 1: (Lcp, Ucp), (Lcp, U cp) = Pivot(Z (1) , 1 Q , γ) 2: Set P = P ∖ (Lcp ∪ U cp) 3: Set Z ′ = (Z (2) ( P ), Z (3) ( P ), Z (4) ( P ), Z (5) (P )) be the sequence of reduced samples where the three first samples are restricted to P . To finish defining DoubleTrisection, we simply need to plug a dimension reduction procedure to select a subset of questions Q ⊂ [d] and then to sum the data on these questions.

The two following algorithms are mainly definitions. For some data

Y ∈ R [n]×[d]
, a set of experts P and a set of blocks Q ⊂ Q r , and a scale r, the P × Q matrix Encode -Matrix(Y, P , Q, r) is simply a reduced matrix where we consider the normalized row sums of Y around the questions of Q at scale r.

Algorithm 7 Encode -Matrix(Y, P , Q, r)

Require: A matrix Y ∈ R [n]×[d]
, a set of experts P and a set of blocks Q ⊂ Qr, a scale r Ensure:

A reduced matrix Z ∈ R P ×Q 1: for i ∈ P and l ∈ Q do 2: Define Z i,l = 1 √ r ∑ k∈[l,l+r) Y i,k ▷ Y i,k = 1 for k ≥ d + 1 3: end for 4: return Z ∈ R P ×Q
▷ the restriction of Z to P and Q Second, Encode -Set(D, r) transforms a subset [d] of questions into a subset Q ⊂ Q r of blocks of questions at scale r.

Algorithm 8 Encode -Set(D, r)

Require: A set of questions D ⊂ [d], a scale r ∈ R Ensure: A set of blocks Q ⊂ Qr return Q = {l ∈ Qr ∶ [l, l + r) ∩ D ≠ ∅}
Then, we are in position to redefine this version of Algorithm 3. As in the original definition in Section 3, there are two variations of this procedure depending on whether we are building the estimator πHT or the estimator πW M that uses the memory of the tree. Algorithm DoubleTrisection(Y, T , P , γ) takes some original data and then reduces the dimension of the problem to build two trisections of the set P of experts.

Algorithm 9 DoubleTrisection(Y, T , P , γ)

Require: 6 samples Y = (Y (1) , . . . , Y (6) ), a tree T , a set of expert P included in a leaf G of T at maximal depth and a pivot index γ Ensure: Two couples of subsets (L, U ), (L, U ) of P

1: Initialize L, U, L, U = ∅ 2: for h ∈ H, r ∈ R do 3:
if Not using the memory of the tree then 4:

Set Q ∶= Qcp(h, r) = DimensionReduction(Y (1) , P , h, r) -see Algorithm 12 or (20) 5:

else if Using the memory of the tree then 6:

Set Q ∶= QW M (h, r) = DimensionReduction -WM(Y (1) , T , P , h, r) -See Algorithm 13 or (26) 7:

end if 8:

Consider the five samples Y ′ = (Y (2) , Y (3) , Y (4) , Y (5) , Y (6) ) 9:

Consider the five reduced samples Z = Encode -Matrix(Y ′ , P , QW M , r) 10:

Compute

(L loc , U loc ), (L loc , U loc ) = DoubleTrisection -Local(Z, γ) 11: Update L = L ∪ L loc and L = L ∪ L loc and U = U ∪ U loc and U = U ∪ U loc 12: end for 13: return (L, U ), (L, U )
Finally, we reproduce BlockSort here that was originally defined in Algorithm 2. We recall that BlockSort iteratively applies a logarithmic number of times the procedure DoubleTrisection to build two suitable trisections of a set G of experts. Although implicit in this description, there are two different versions of the corresponding procedure whether we use the memory of the tree -estimator πW M -or not -estimator πHT in DoubleTrisection. In the following, τ ∞ = ⌈4 ⋅ 10 7 log 7 ( nd δ(ζ-) 2 )⌉ stands for the number of iterations in BlockSort. (Lτ , Uτ ), (Lτ

, U τ ) = DoubleTrisection(Yτ , T , G ∖ (Oτ ∪ Iτ ), γ) as in Algorithm 9 6: Update Oτ+1 = Oτ ∪ Lτ , Iτ+1 = Iτ ∪ Uτ , Oτ+1 = Oτ ∪ Lτ , Iτ+1 = Iτ ∪ U τ 7: end for 8: if Oτ ∞ ∩ Iτ ∞ ≠ ∅ then 9: Set Oτ ∞ ∶= Oτ ∞ ∖ Iτ ∞ and Iτ ∞ ∶= Iτ ∞ ∖ Oτ ∞ 10: end if 11: return (Oτ ∞ , G ∖ (Oτ ∞ ∪ Iτ ∞ ), Iτ ∞ )
Under an event of high probability (to be later discussed), we have O τ∞ ∩ I τ∞ = ∅. The correction at the end of the algorithm simply forces the algorithm to return a partition of G.

A.2. Hierarchical sorting Trees and TreeSort Algorithm

In this subsection, we formally describe how we build and navigate into a hierarchical tree. In the following, a node G ∈ Nodes is a labelled subset of [n]. Its label belongs to {0, p, 1}. For a node G, we write Type(G) for the label (also called type) of G. Definition 1. (Hierarchical sorting Trees) A hierarchical sorting tree T is a rooted tree that satisfies the three following properties:

• The root G of T corresponds to the set [n] and its label is 0.

• Any node G of type p is a leaf.

• Any node G of type in {0, 1} is either a leaf or has three children (O, P, I) with type 0, p, 1 respectively. Besides, (O, P, I) correspond to a partition of G.

We write T 0 for the tree of depth 0. The procedure TreeSort iteratively builds a hierarchical sorting tree. Hence, we need to define the operation of adding children to a leaf in a tree T . For a specific leaf G of type 0 or 1, we consider three labelled subsets O, P , I of type 0, p, 1, respectively. Besides, those subsets satisfy the third condition in Definition 1. Then, T ′ = AddChild(T , G, (O, P, I)) is the supertree of T where we have added the nodes (O, P, I) as children of G. Finally, we observe that for any t > 0, all the nodes at depth t of a hierarchical sorting tree T are disjoint.

In fact, we shall prove in Proposition B.1 and in Corollary B.4 that, with high probability, the final tree T t∞ turns out to be a valid hierarchical sorting tree as defined below. The second property ( O < G and I < G ) forces the tree to be finite. For a node G in a such valid hierarchical sorting tree T , Depth(T , G) stands for the depth of G in T . In light of this definition of valid hierarchical sorting trees, a labelled subset G cannot appear twice in a tree T , so that Depth(T , G) is well-defined.

We are now equipped to provide a more formal definition of TreeSort, although the procedure is in fact the same as the one described in Algorithm 1. Let t ∞ = ⌈log(n) log(2)⌉.

Algorithm 11 TreeSort(Y)

Require: 6τ∞t∞ samples Y = (Y (0) , . . . , Y (6τ∞t∞-1) ) Ensure: A final tree T 1: T = T0 ▷ The root is at depth 0 2: for t = 0, . . . , t∞ -1 do 3:

Consider 6τ∞ fresh samples Y = (Y (6tτ∞) , . . . , Y (6(t+1)τ∞-1) ) 4:

for G ∈ L (0,1) (T ) do ▷ See (39) for the definition of L (0,1) 5:

(O G , P G , I G ) = BlockSort(Y, T , G) 6:
Set Type(O G ) = 0 and Type(P G ) = p and Type(I G ) = 1 7:

end for 8:

for G ∈ L (0,1) (T ) do 9:

AddChild(T , G, (O G , P G , I G )) 10:
end for 11: end for 12: return T As explained in Section 3, the final estimators πHT or πW M are computed from their corresponding hierarchical sorting tree T .

In order to define the DimensionReduction -WM algorithm in the next subsection, we need to introduce a few more notation. First, we define

L (0,1) (T ) = {G ∈ Leaves(T ) ∶ Type(G) ∈ {0, 1}} , (39)
as the collection of leaves of T that are either of type 0 or of type 1. In the algorithm TreeSort, these are the leaves to be partitionned. In particular at step t of TreeSort, L (0,1) (T ) is only made of leaves at depth t.

For a subset P ⊂ [n], Leaf (T , P ) is defined as the leaf G ∈ Leaves(T ) containing P (if it exists). Finally, the groups G ∈ L (0,1) (T ) inherit from a natural order provided that T is a valid hierarchical sorting tree. We can enumerate the groups G 1 , G 2 ,. . . , G L (0,1) (T ) in such a way that all the experts in G s are below those of G s ′ for s < s ′ . To ease the presentation, we also introduce, for any positive integer s the groups G L (0,1) (T ) +s = {n + s}. The corresponding data and signal for the n+s-th expert satisfies Y n+s,j = 1 = M n+s,j almost-surely for any j ∈ [d]. Also, for any positive integer s we introduce the groups G 1-s = {1 -s}. The corresponding data and signal for this synthetic expert satisfy Y 1-s,j = 0 = M 1-s,j = 0 almost-surely for any j ∈ [d].

Then, for a specific leaf G s ∈ L (0,1) (T ), Order(T , G) stands for the collection (G (a) ), a ∈ Z of leaves where G (a) = G a+s . In other words, we have G (0) = G s and G (1) is the following group, and so on.

A.3. Dimension Reduction Algorithms

To finish the description of the two procedures, we fully describe the two dimension reduction algorithms both for the oblivious estimator πHT and for the estimator πW M with memory. These procedures were already introduced in Section 3. First, DimensionReduction(Y, P , h, r) considers the columns-wise mean of the restriction of Y to the group P and detects highvariation regions of this vector.

Algorithm 12 DimensionReduction(Y, P , h, r) For the more involved dimension reduction procedure with memory DimensionReduction -WM, we compute the CUSUM statistic in larger groups V ⊃ P to reduce its variance and we also require that the estimated "width" of the group of experts is high enough. More precisely, given three sets of expert V, V + and V -and a sample Y , we consider the two following statistics, for any k = 1, . . . , d and r ′ ∈ R:

Require: A sample Y ∈ R P ×[d]
∆(ext) k,r ′ (V + , V - ) = 1 2r ′ k+r ′ -1 k ′ =k-r ′ y k ′ (V + ) -y k ′ (V - ) ; Ĉ(ext) k,r ′ (V) = 1 r ′ ⎛ ⎝ k+r ′ -1 k ′ =k y k ′ (V) - k-1 k ′ =k-r ′ y k ′ (V) ⎞ ⎠ . (41)
Here, ∆(ext) k,r ′ (V + , V -) computes the width -i.e. the difference -between the mean of experts in V + and the mean of experts in V -. Since V + and V -are built in the algorithm below in such a way that experts in P are below those of V + and above those of V -, ∆(ext) k,r ′ (V + , V -) provides an upper bound of the width between the best expert in P and the worst expert in P .

The algorithm DimensionReduction -WM described below builds a collection of sets V + , V -, and V and detects questions such that both the CUSUM C (ext) k,r ′ (V) and the width ∆(ext) k,r ′ (V + , V -) are large enough. Further explanations are postponed to the analysis of the algorithm in Section F. Below, we write ⌈x⌉ dya for 2 ⌈log 2 (x)⌉ .

Algorithm 13 DimensionReduction -WM(Y, T , P , h, r)

Require: A sample Y ∈ R n×d , a tree T , a set P included in a leaf G of T of type 0 or 1, h ∈ H and r ∈ R Ensure: A set of blocks QW M ∶= QW M (Y, T , P , h, r) ⊂ Qr Appendix B: Overview and organization of the proofs of Theorems 2.2 and 2.3

1: r0 = 2 9 log(4d R δ) ζ 2 P h 2 and r = 4(⌈r0⌉ dya ∨ r) 2: (G (a) ) a∈Z = Order(T , G) 3: for rcp ∈ [4r, 2r] ∩ R do 4: Set a + W M = min{a ∶ G (1) + ⋅ ⋅ ⋅ + G (a) ≥ 2 11 log(4d R δ) ζ 2 rcph 2 } 5: Set a - W M = min{a ∶ G (-1) + ⋅ ⋅ ⋅ + G (-a) ≥ 2 11 log(4d R δ) ζ 2 rcph 2 } 6: Set V + rcp ∶= V + rcp (T , G, h) = a + W M ⋃ a=1 G (a) and V - rcp ∶= V - rcp (T , G, h) = -1 ⋃ a∈-a - W M G (a) (42
In this section, we divide the analysis of the procedures into several properties that will be proved to hold with high probability in the next sections.

B.1. Definitions

Since we build our estimator using a hierarchical tree, we need to quantify the error that we suffer at each depth of the tree. For i ∈ [n], we write M i = M i,⋅ for the expert i. By definition of π * , we recall that

M π * -1 (1) ≤ M π * -1 (2) ≤ . . . ≤ M π * -1 (n) .
For a given group of experts G, we write π * {G} for the oracle ordering in [1, G ] of the group G according to π * , that is for all i, j ∈ G, π * {G} (i) and π * {G} (j) belong to [1, G ] and

π * {G} (i) < π * {G} (j) iff π * (i) < π * (j) .
We say that a sequence of sets G = (G 1 , . . . , G α ) is an ordered partition of a set S if {G 1 , . . . , G α } is a partition of S. For a given ordered partition {G 1 , . . . , G α } and a ∈ [1, α] and any i ∈ G a we write

π - G (G a ) = π - G (i) ∶= a ′ <a G a ′ and π + G (G a ) = π + G (i) ∶= a ′ ≤a G a ′ . (43)
If we are to build a permutation π which is consistent with this ordered partition, then one easily checks that π(i)

∈ [π - G (i) + 1, π + G (i)].
For simplicity, we write G(i) for the group G a such that i ∈ G a . For a given ordered partition G = (G 1 , . . . , G α ), we define the oracle permutation associated to G by

π * G (i) = π - G (i) + π * {G(i)} (i) . (44)
For example, π * {[n]} = π * is simply the true permutation. By definition, we have

π * G (G(i)) = [π - G (i) + 1, π + G (i)]
. Given an ordered partition, π * G is the best permutation we could hope for after any statistical treatment.

Given an ordered partition G = (G 1 , . . . , G α ), we define the random estimation of π * given

G as πG (i) which is uniformly distributed in [π - G (i) + 1, π + G (i)]: πG (i) ∈ [π - G (i) + 1, π + G (i)]
. Note that πG is not necessarily bijective.

B.2. Deterministic Analysis

In this subsection, we analyze TreeSort (Algorithm 11) and we characterize the loss of the estimator π in terms of that of the trisections that are computed inside the subroutine BlockSort(Y, T , G). This algorithm takes a subset G of experts and computes two trisections of G. The first one The last claim states that all experts in O are below all experts of I. The second claim can be understood as the fact that, if an expert i belongs to O, then all experts below i belong to O.

Let Y = (Y (0) , . . . , Y (6τ∞-1) ) be a sequence of 6τ ∞ matrices in R n×d . We say that BlockSort satisfies Property 1 on (Y, T , G) if the two partitions (O, P, I) and (O, P , I) worked out in Algorithm 11 satisfy Property 1. We recall that by definition O ⊂ O, I ⊂ I, P ⊂ P so that P corresponds to the collection of experts that are either not sorted by TreeSort or are sorted with a small confidence.

For each t = 0, . . . , t ∞ , we write T t for the hierarchical sorting tree at the beginning of step t of TreeSort. Besides, we write G t for the corresponding ordered partition obtained by taking the leaves of the tree T t in increasing order in the ternary base {0, p, 1}. We define the tree T t∞ as the tree T t∞ where we replaced all the leaves P of type p -at any depth -by P , where (O, P , I) has been worked out by TreeSort at the same time as (O, P, I).

We also define

L t (T t∞ ) = {P ∈ G t∞ ∶ P is a nonempty leaf at depth t of T t∞ } ; L t (T t∞ ) = {P ∈ G t∞ ∶ P is a nonempty leaf at depth t of T t∞ } . ( 45
)
For simplicity, we sometimes write L t for L t (T t∞ ) and L t for L t (T t∞ ). L t stands for the collection of experts that have not been sorted at the t-th iteration TreeSort. The sets in the collection L t are strictly larger and correspond to the collections of experts in P that are either not sorted by TreeSort or are sorted with less confidence. Let M (P ) be defined as the restriction of M to the experts in P , and M (P ) the P ×d matrix with constant columns which correspond to the mean row of M (P ). The following proposition characterizes the loss of the final estimator estimator πGt ∞ which is obtained from a hierarchical sorting tree in terms of the variance of the experts M within the groups P in L t (T t∞ ).

Proposition B.1 (Deterministic Analysis of TreeSort). Assume that at each step of TreeSort, the routine BlockSort applied to the data satisfies Property 1. Then, the error of

π = π-1 Gt ∞ is controlled as follows M π-1 -M π * -1 2 F ≤ 10t ∞ t∞ t=1 P ∈Lt M (P ) -M (P ) 2 F . (46)
Besides, the hierarchical tree T t∞ is valid (as in Definition 2) and all its non-empty leaves are of type p.

Up to a normalization, M (P ) -M (P ) 2 F corresponds to the variance of M within the group P . The bound (46) expresses that the loss of a hierarchical sorting tree is controlled by the variance of the set P that are not sorted with confidence at each step of the algorithm. Also, we recall that t ∞ = ⌈log(n) log(2)⌉. This proposition only relies on Property 1 and on the construction of the tree. Hence, it applies both to the estimators πHT and πW M . In light of Propositions B.1 and B.2, it suffices to show that, with high probability, all applications of DoubleTrisection in the construction of the hierarchical sorting tree satisfy Property 2, and then to control the sum of within-group variances in (46).

B.3. High probability Control of Property 2

We write in this part of the proof (this subsection), for simplicity, Y = (Y (1) , . . . , Y (6) ) for 6 independent matrices that are identically distributed as Y = M + E in [START_REF] Aldous | Exchangeability and related topics, École d'été de probabilités de Saint Flour[END_REF], where we recall that the entries of E are centered, independent and ζ-subgaussian.

Fix a hierarchical sorting tree T (recall Definition 1), a leaf G of T , a set P ⊂ G, a pivot γ ∈ {1, . . . , P }. Let P 2 ∶= P 2 (T , P , γ, β tris , β tris ) be the event holding true if DoubleTrisection satisfies Property 2 on Y for (T , P , γ, β tris , β tris ). The following proposition states that P 2 holds with uniformly high probability.

Proposition B.3. For any T , any leaf G, any P ⊂ G, any pivot γ ∈ [ P ], we have

P(P 2 ) ≥ 1 -3 H R δ.
This result is valid for both versions of DoubleTrisection where we use the memory of the tree (estimator πW M ) or not (estimator πHT ). Recall that in BlockSort there are at most τ ∞ calls of DoubleTrisection. Since the construction of the hierarchical tree requires at most 2 t∞+1 applications of BlockSort, we arrive at the following straightforward corollary of Propositions B.2, B.1 and B.3.

Corollary B.4.

There exists an event ξ of probability higher than 1 -2 t∞+1 3τ ∞ H R δ such that all results of BlockSort within TreeSort satisfy Property 1. In particular, the tree T t∞ is a valid hierarchical sorting tree (as in Definition 2) whose non-empty leaves are all of type p. Besides, on this event we also have

M π-1 G t∞ -M π * -1 2 F ≤ 10t ∞ t∞ t=1 P ∈Lt M (P ) -M (P ) 2 F . (47)
Again, this results applies to both variants of our procedure -with or without memory.

B.4. Control of the Loss function

In contrast to the previous subsection, we now need to specify the dimension reduction scheme DimensionReduction (which corresponds to πHT ) or DimensionReduction -WM (which corresponds to πW M ) inside DoubleTrisection as the convergence rates depend on these quantities.

First we state the results for the method without memory: πHT .

Proposition B.5. Consider the oblivious hierarchical sorting tree estimator πHT . On the intersection of event ξ (defined in Corollary B.4) and an event of probability higher than

1 -5 ⋅ 2 t τ ∞ δ, it holds that P ∈Lt M (P ) -M (P ) 2 ≲ ζ 2 log 8.5 6nd δζ - n 2 3 d 1 3 ζ 2 3 ∧ nd 1 6 ζ 1 3 ∧ n √ d + n .
Then we state the results for the method with memory: πW M .

Proposition B.6. Consider the hierarchical sorting tree estimator πW M . On the intersection of event ξ (defined in Corollary B.4) and an event of probability higher 1 -5 ⋅ 2 t τ ∞ δ, it holds that

P ∈Lt M (P ) -M (P ) 2 ≲ ζ 2 log 9 6nd δζ - n 3 4 d 1 4 ζ 1 2 ∧ nd 1 6 ζ 1 3 ∧ n √ d ∧ n 2 3 √ d ζ 1 3 + n .
Now, we are in position to easily conclude the proof of Theorems 2.2 and 2.3.

Proof of Theorem 2.2. Let πHT ∶= πGt ∞ denote the oblivious hierarchical sorting tree estimator. Combining Corollary B.4 with Proposition B.5 and a union bound over all t = 0, . . . , t ∞ -1, it holds with probability higher than

1 -8 ⋅ 2 t∞+1 τ ∞ H R δ that M π-1 HT -M π * -1 2 F ≲ t 2 ∞ ζ 2 log 8.5 2nd δζ - n 2 3 d 1 3 ζ 2 3 ∧ nd 1 6 ζ 1 3 ∧ n √ d + n ≲ ζ 2 log 10.5 2nd δζ - n 2 3 d 1 3 ζ 2 3 ∧ nd 1 6 ζ 1 3 ∧ n √ d + n .
Proof of Theorem 2.3. Let πWM ∶= πGt ∞ denote the hierarchical sorting tree where we use the memory to reduce the dimension (Algorithm DimensionReduction -WM). Combining Corollary B.4 with Proposition B.6 and a union bound on t = 0, . . . , t ∞ -1, it holds with probability higher than

1 -8 ⋅ 2 t∞+1 τ ∞ H R δ that M π-1 W M -M π * -1 2 F ≲ ζ 2 log 11 6nd δζ - n 3 4 d 1 4 ζ 1 2 ∧ nd 1 6 ζ 1 3 ∧ n √ d ∧ n 2 3 √ d ζ 1 3
+ n .

In the next four sections, we prove the intermediary results. Proof of Proposition B.1. First, we prove by induction that T t∞ is a valid hierarchical sorting tree. Besides, the last part of Property 1 enforces that the cardinality of any non-terminal node G of T t∞ of depth t is at most n 2 t . As a consequence, the cardinality of any non-terminal node at depth t ∞-1 is at most 1 and its children O and I are therefore empty.

We control the error using a telescopic sum. Recall that, by convention, π * G 0 = π * . We start with the following inequality:

M π-1 G t∞ -M π * -1 2 F ≤ 2 M πG t∞ -M π * -1 G t∞ 2 F + 2t ∞ t∞ t=1 M π * -1 G t -M π * -1 G t-1 2 F . (48)
Since, for any group P in G t∞ , πGt ∞ sorts the elements of P uniformly at random and π * -1

Gt ∞ acts as another permutation of P , we deduce from the triangular inequality that

M π-1 G t∞ -M π * -1 G t∞ 2 F ≤ P ∈Gt ∞ 2 M (P ) -M (P ) 2 F = 2 t∞ t=1 P ∈Lt M (P ) -M (P ) 2 F ≤ 2 t∞ t=1 P ∈Lt M (P ) -M (P ) 2 F , (49)
where we used in the last line that P ⊂ P . For the second term in (48), remark that π * G t-1 (P ) = π * Gt (P ) for any P ∈ L t-1 so that the error at step t in the telescopic sum can be restricted to the groups G that are trisected at step t -1:

t∞ t=1 M π * -1 G t -M π * -1 G t-1 2 F = t∞ t=1 G∈G t-1 ∖(∪ s≥1 Lt-s) i∈G M π * -1 G t (π * G t-1 (i)) -M i 2 2 .
Let (O, P, I) be the trisection obtained at the t-th iteration when we apply BlockSort to a group G ∈ G t-1 ∖ (∪ s≥1 L t-s ). We also write (O, P , I) for the more conservative trisection obtained at the end of BlockSort. For short, we write ω = π * -1 Gt π * G t-1 . We decompose the sum over i ∈ G:

i∈G M ω(i) -M i 2 = i∈O M ω(i) -M i 2 + i∈I M ω(i) -M i 2 + i∈P M ω(i) -M i 2 ,
By Property 1, all the experts in O and in I are perfectly sorted within G by π * -1 G(t) . As a consequence, the two first sums in the right-hand side term of the above equality are equal to zero. To handle the last term, we introduce the row vector m(P ) as the mean of the experts of M over P :

i∈G M ω(i) -M i 2 2 = i∈P M ω(i) -M i 2 2 ≤ 2 i∈P ( M i -m(P ) 2 2 + m(P ) -M ω(i) 2 2 ) = 4 M (P ) -M (P ) 2 F ,
where we used in the last line that ω acts as a permutation of P . Since P ∈ L t , we obtain

t∞ t=1 M π * -1 G t -M π * -1 G t-1 2 ≤ 4 t∞ t=1 P ∈Lt M (P ) -M (P ) 2 F .
Together with (48) and (49), this concludes the proof since t ∞ ≥ 1.

Proof of Proposition B.

O τ = ⋃ τ ′ <τ L τ ′ and I τ = ⋃ τ ′ <τ U τ ′ and O τ = ⋃ τ ′ <τ L τ ′ and I τ = ⋃ τ ′ <τ U τ ′ ,
we easily deduce from Property 2 for (L τ , U τ ) and (L τ , U τ ) that the first part of Property 1 is satisfied for (O τ , P τ , I τ ), and (O τ , P τ , I τ ).

Let us turn to the third and fourth parts of Property 1. Let us call i m the expert such that π * {G} (i m ) = ⌊ G 2⌋. In fact, we only need to prove that max i∈Oτ π * {G} (i) ≤ G 2 and min i∈Iτ π * {G} (i) ≥ G 2. For this purpose, we prove by induction on τ that the pivot always satisfies π * -1 {P τ } (γ) = i m and that all the experts of O τ (resp. I τ ) are below (resp. above) i m , where γ depends on τ and is defined in Algorithm 10. Assume that this property holds at step τ . Since O τ only contains experts that are below the median expert and since

γ = ⌊ G 2⌋-O τ , it follows that π * -1 {P } (γ) = i m . Consider any i ∈ O τ +1 . If i ∈ O τ , then π * {G} (i) ≤ G 2 by induction. If i ∈ L τ ,
then it follows from Property 2 that i is below i m , which in turn implies that π * {G} (i) ≤ G 2. By symmetry, the property also holds for I τ . We have proved the third and the fourth parts of Property 1.

Finally, we consider the second part of Property 1. Assume that the property holds at step τ . This implies that, for any i ∈ O τ , all experts below i belong to O τ . Consider any expert

i ∈ O τ +1 . If i ∈ O τ , then, by induction, we have π * {G} (i) = π * {Oτ } (i) = π * {O τ +1 } (i).
Then, we turn to the case where i belongs to L τ ⊂ P τ . Consider any j ∈ G such that π * {G} (j) ≤ π * {G} (i). If j ∈ O τ , then we obviously have j ∈ O τ +1 . If j ∈ P τ , then the second part of property 2 enforces that j ∈ L τ and therefore j ∈ O τ +1 . Finally, it is not possible that j ∈ I τ since this enforces that π * {G} (j) > G 2 > π * {G} (i) and contradicts the hypothesis. We prove similarly that, for any expert i ∈ I τ +1 , all experts j above i belong to I τ +1 .

Proof of Proposition B.3. As DoubleTrisection is based on multiple applications of the Pivot algorithm, we start by considering the latter procedure.

Consider two sets P ⊂ [n] and Q ⊂ [d] and a matrix Θ ∈ R P × Q which, up to the permutation π * {P } , is bi-isotonic. Let Z be a noiy observation of Θ,

Z = Θ + N , (50)
where the noise matrix N is made of independent, centered, ζ-subGaussian random variables. Let w ∈ R Q + be a non-zero vector with nonegative coordinates. we write (L, U ) and (L, U ) for the result of Pivot(Z, w, γ). We define the event P 3 ∶= P 3 (P , Q, w, γ) as the event on Z such that (L, U ) and (L, U ) satisfy Property 2.

We remind that P ′ = P ∖ (L ∪ U ), and P ′ = P ∖ (L ∪ U ).

Lemma C.1. For any non-zero vector w ∈ R Q + , any pivot γ ∈ {1, . . . , P } , we have P[P 3 ] ≥ 1 -δ. Besides, on the same event of probability at least 1 -δ, we have

⟨Θ i,⋅ -Θ iγ ,⋅ , w w 2 ⟩ ≤ (2ζ √ 2 + β tris ) log 2 P δ if i ∈ P ′ . (51)
Before proving the lemma, let us explain why Proposition B.3 is easily deduced from it. The procedure DoubleTrisection calls at most 3 H R times Pivot. Note that, each time, we rely on an independent sample to choose the direction w and to apply Pivot. Then, applying the Lemma, we derive that, with probability higher than 1 -3 H R δ, each of these 3 H R sets (L, U ) and (L, U ) satisfy Property 2. Hence, we only need to check that Property 2 is stable by union. If, both (L (1) , U (1) ) and (L (1) , U

) and (L (2) , U (2) ) and (L (2) , U

) satisfy Property 2, then one easily checks that the first and third part of Property 2 are also true for

(L, U ) = (L (1) ∪ L (2) , U (1) ∪ U (2)
) and (L, U ) = (L

∪ L (2) , U

∪ U

). Consider any expert i in L. Without loss of generality, we may assume that i ∈ L (1) so that all experts below i in P belong to L (1) by the second part of Property 2. As a consequence, all these experts below i belong to L and we deduce that the second part of Property 2 holds. Similarly, we deal with experts i ∈ U . This concludes the proof of Proposition B.3.

Proof of Lemma C.1. Since the noise matrix in (50) is made of independent ζ-subGaussian random variables, it follows from a union bound, that with probability higher than 1 -δ, we have

⟨Z i,⋅ , w w 2 ⟩ -⟨Θ i,⋅ , w w 2 ⟩ ≤ ζ 2 log 2 P δ .
simultaneously for all i in P . Since the entries of w are non-negative, the quantities ⟨Θ i,⋅ , w w 2 ⟩ are ordered according the permutation π * {P } . Denote i γ = π * -1 {P } (γ) and îγ the index of γ-th value of ⟨Z i,⋅ , w w 2 ⟩ for i ∈ P . Since at least γ experts satisfy ⟨Θ i,⋅ , w w 2 ⟩ ≤ ⟨Θ iγ ,⋅ , w w 2 ⟩, we deduce from the above uniform deviation inequality that

⟨Z îγ ,⋅ , w w 2 ⟩ ≤ ⟨Θ iγ ,⋅ , w w 2 ⟩ + ζ 2 log 2 P δ .
By symmetry, we deduce that

⟨Z îγ ,⋅ , w w 2 ⟩ -⟨Θ iγ ,⋅ , w w 2 ⟩ ≤ ζ 2 log 2 P δ .
As a consequence, we have

⟨Θ i,⋅ , w w 2 ⟩ < ⟨Θ iγ ,⋅ , w w 2 ⟩ -(β tris -2ζ √ 2) log 2 P δ if i ∈ L ; ⟨Θ i,⋅ , w w 2 ⟩ > ⟨Θ iγ ,⋅ , w w 2 ⟩ + (β tris -2ζ √ 2) log 2 P δ if i ∈ U ; ⟨Θ i,⋅ -Θ iγ ,⋅ , w w 2 ⟩ ≤ (2ζ √ 2 + β tris ) log 2 P δ if i ∈ P ′ . ( 52
)
The same inequalities hold for L, U , and P ′ provided that we replace β tris by β tris . It remains to show that (L, U ) and (L, U ) satisfy Property 2. The first part of the property is obvious. Since

β tris ≥ 2 √ 2ζ, one observes that π * {P } (i) < π * {P } (i γ ) = γ if i ∈ L. Similarly, π * {P } (i) > γ if i ∈ U
and the third part of Property 2 follows. Turning to the second part of the property, we consider without loss of generality some i ∈ L and we need to show that all j satisfying π * {P } (j) ≤ π * {P } (i) belong to L. First, such a j does not belong to U since π * {P } (i) < γ. Since i ∈ L, we deduce that

⟨Θ i,⋅ , w w 2 ⟩ < ⟨Θ iγ ,⋅ , w w 2 ⟩ -(β tris -2 √ 2ζ) log 2 P δ ,
which implies that

⟨Θ iγ ,⋅ -Θ j,⋅ , w w 2 ⟩ > (β tris -2 √ 2ζ) log 2 P δ ≥ (2ζ √ 2 + β tris ) log 2 P δ .
which in light of (52) implies that j ∉ P ′ . We have proved that j belongs to L. Hence, Property 2 holds, which concludes the proof.

D.1. Notation

As the arguments rely on considering aggregation of the matrix at different scales, we recall some notation. Let Y = M + E denote a sample of the original matrix. For a set P ⊂ [n] of experts and a set Q of blocks of questions and a scale r ∈ R, we respectively denote

Z(P, Q, r) = Encode -Matrix(Y, P, Q, r) ∈ R P ×Q Θ(P, Q, r) = Encode -Matrix(M, P, Q, r) N (P, Q, r) = Encode -Matrix(E, P, Q, r) ,
the aggregations of Y , M , and E at scale r so that

Z(P, Q, r) = Θ(P, Q, r) + N (P, Q, r) .
By definition of Encode -Matrix, all the entries of N are independent and ζ-subGaussian.

For any p × q matrix A, we define a as the row vector corresponding to the column-wise mean of A, that is a j = 1 q ∑ q i=1 A i,j . Besides, we write A for p × q matrix whose experts are all equal to a.

D.2. Analysis of DimensionReduction

In this subsection, we mainly state, that for any h ∈ H, r ∈ R, the set Qcp (which depends on h, r) detects the high-variation regions of M with high probability. Then, we show in Lemma D.2, that for, for some (h, r) ∈ H × R, the aggregation of M at scale r and at these high-variation regions contains most of the variance of M . This motivates us to work with this aggregated matrix henceforth. Consider any set P of experts and a sample Y (P ) = M (P ) + E(P ) .

Fix any scale r ∈ R and any height h ∈ H. Recall the two quantities r 0 and r defined in DimensionReduction by

r 0 = 32ζ 2 log 2d δ 1 P h 2 and r = 8(⌈r 0 ⌉ ∨ r) . (53)
In a nustshell, r 0 stands for the minimal scale at which a variation of order h in the mean m(P ) = E[y(P )] can be statistically detected. This is why we consider empirical variations of y(P ) at scale r ≥ (r 0 ∨ r) in Algorithm DimensionReduction to possibly detect variations at scale r. The purpose of this subsection is to prove, that with high probability, the collections Qcp (h, r) = DimensionReduction(Y, P , h, r) of selected blocks of length r is not too large and that there exists at least one (h, r) ∈ H × R such that the aggregation of M (P ) at scale r restricted to the blocks Qcp (h, r) captures most of the variance of M (P ).

For this purpose, we recall the CUSUM statistics introduced in DimensionReduction and we introduce its population counterpart. Given positive integers k ∈ [d] and r > 0, consider

Ĉk,r = 1 r k+r-1 k ′ =k y k ′ (P ) - k-1 k ′ =k-r y k ′ (P ) and C * k,r = 1 r k+r-1 k ′ =k m k ′ (P ) - k-1 k ′ =k-r m k ′ (P ) .
Equipped with this notation, we define Dcp as in the algorithm as the set of positions d such that the association CUSUM statistic is above the threshold, and D * cp and D * cp as some population versions of Dcp , but with different tuning parameters:

Dcp (h, r) = k ∈ [d] ∶ Ĉk,r ≥ 1 4 h , (54)
D * cp (h, r) = k ∈ [d] ∶ C * k,8r ≥ 1 2 h ; D * cp (h, r) = k ∈ [d] ∶ C * k,r ≥ 1 8 h . (55)
Then, we consider the collection of blocks Qcp (h, r) , Q * cp (h, r), and Q * cp (h, r) of size r that are associated with these positions. In terms of our algorithms, this means that

Q * cp (h, r) = Encode -Set(D * cp , r), Q * cp (h, r) = Encode -Set(D * cp , r
), and Qcp (h, r) = Encode -Set( Dcp , r). The first proposition states that, with high probability, Qcp (h, r) is sandwidched between Q * cp (h, r) and Q * cp (h, r), so that, on the corresponding event, it is sufficient to study these two quantities.

Lemma D.1. For all h, r, the event ξ cp ∶= ξ cp (P , h, r) defined by

Q * cp ⊂ Qcp ⊂ Q * cp , (56)
holds true with probability at least 1 -δ.

Then, we show that there are not too many significant blocks in Q * cp . The proof is based on the fact that the row vector m(P ) is isotonic and lies in [0, 1]. As a consequence, there cannot exist two many regions where the variations of m(P ) is large.

Lemma D.2. For all h ∈ H and all r ∈ R, we have

Q * cp ≤ 64r rh . (57)
The next lemma states that, at least for a height h ∈ H and a scale r ∈ R, the aggregation of M at scale r and restricted to the regions Q * cp (h, r) of significant variations contains almost all the variance of the signal.

For any number θ and any η > 0, we define [θ] η = (-1) sgn(θ) η1 θ ≥η . For any matrix Θ, we write [Θ] η for the thresholded matrix with coefficients [Θ i,j ] η .

Lemma D.3. For any set P ⊂ [n] and any bi-isotonic matrix M ∈ [0, 1] n×d , there exist r ∈ R and h ∈ H such that

M (P ) -M (P ) 2 F ≤ 16ζ 2 + 96 R H Θ(P , Q * cp ) -Θ(P , Q * cp ) √ rh 2 F , (58)
The proof of the above lemmas is postponed to Appendix E.1.

D.3. Analysis of Pivot based on the row sums

We consider a specific subset P of experts, a subset Q of blocks of questions, the corresponding aggregated model

Z(P , Q) = Θ(P , Q) + N (P , Q) ∈ R P ×Q , (59)
and a pivot γ ∈ [1, P ]. Let (L, U ) = Pivot(Z, 1 Q , γ) be the conservative result of Pivot based on the row sums of Z(P , Q) and let P ′ = P ∖ (L ∪ U ) be the subgroup of experts which have not been classified by Pivot. The following proposition states that, provided that for some η the norm Θ(P , Q) -Θ(P , Q) η 2 F is large enough compared to P Q , the resulting matrix Θ(P ′ , Q) -Θ(P ′ , Q) after Pivot has a significantly smaller norm. We shall often use the following quantity.

φ l 1 = 2(2ζ √ 2 + β tris ) ≤ 29ζ . ( 60
)
Proposition D.4. Consider any P ⊂ [n], any r ∈ R, and any subset Q ⊂ Q r . Also, fix any η > 0 and any φ > 0. If

Θ(P , Q) -Θ(P , Q) η 2 F ≥ 1 φ Θ(P , Q) -Θ(P , Q) 2 F ≥ 8φ l 1 η log( 2 P δ ) P Q ,
then, with probability higher than 1 -δ, we have

Θ(P ′ , Q) -Θ(P ′ , Q) 2 F ≤ 1 - 1 16φ Θ(P , Q) -Θ(P , Q) 2 F .

D.4. Analysis of DoubleTrisection -PCA

In this subsection, we state the main result regarding the trisection of a set P based on the first singular vector of a suitable matrix. We start from a subset P of experts. In DoubleTrisection -Local, we start applying Pivot and define P = P ∖(L cp ∪U cp ) as the set of experts that have not been classified by Pivot. We are given four independent samples Z = (Z (1) , Z (2) , Z (3) , Z (4) ) according to the aggregated model (59). The first three samples are restricted to P , whereas the last one concerns P . Fix γ ∈ [1, P ]. We consider (L, U ) = DoubleTrisection -PCA(Z, γ) and P ′ = P ∖(L pca ∪U pca ) the set of experts that have not been classified by DoubleTrisection -PCA.

Recall the definition (60) of φ l 1 . Henceforth, the matrix Θ( P , Q) is said to be undistinguishable in l 1 -norm if it satisfies

max i,j∈P Θ i,⋅ ( P , Q) -Θ j,⋅ ( P , Q) 1 ≤ φ l 1 Q log 2 P δ . (61)
Since, up to permutation of its experts, the matrix Θ( P , Q) is bi-isotonic, the l 1 norm Θ i,⋅ ( P , Q)-Θ j,⋅ ( P , Q) 1 is simply the difference of the row sums of Θ( P , Q). Since P has been deduced from P by applying Pivot(Z, γ), we can safely assume that Θ( P , Q) is undistinguishable in l 1 -norm with high probability -see the next subsection for a proper justification.

The next result states that, if Θ( P , Q) is undistinguishable in l 1 -norm and if the Frobenius norm of Θ( P , Q) -Θ( P , Q) is large enough, then the corresponding matrix Θ(P ′ , Q) obtained after trisection has a significantly smaller Frobenius norm.

Proposition D.5. Let P ⊂ [n] and Q ⊂ [d]. If Θ( P , Q) is undistinguishable in l 1 -norm and if Θ( P , Q) -Θ( P , Q) 2 F ≥ 2 ⋅ 10 5 ζ 2 log 3 6nd δζ - P Q + P , (62)
then, with probability higher than 1 -3δ, we have

Θ(P ′ , Q) -Θ(P ′ , Q) 2 F ≤ 1 - 1 200 log 2 (nd ζ -) Θ(P , Q) -Θ(P , Q) 2 F .
Then, we gather the two previous results to analyze the routine DoubleTrisection -Local. Fix any P ⊂ [n] and Q ⊂ [p]. Let Z = (Z (1) (P , Q), Z (2) (P , Q), Z (3) (P , Q), Z (4) (P , Q), Z (5) (P , Q)) be five independent samples of the model (50). Fix any γ ∈ [1, P ]. Let (L, U ) be the conservative result of DoubleTrisection -Local(Z, γ) and P ′ = P ∖ (L ∪ U ). In the following, we write Θ(P , Q r ) for the aggregation of M (P ) at all blocks of size r.

Corollary D.6. Fix any r ∈ R. If, for some P ⊂ [n], Q ⊂ Q r , and η > 0, Θ(P , Q) satisfies

⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ Θ(P , Q) -Θ(P , Q) η 2 F ≥ 1 120 log 2 nd δζ- Θ(P , Q r ) -Θ(P , Q r ) 2 F Θ(P , Q) -Θ(P , Q) 2 F ≥ 4 ⋅ 10 5 ζ 2 log 3 6nd δζ- η ζ P Q ∧ P Q + P , (63)
then, with probability higher than 1 -4δ, we have

Θ(P ′ , Q r ) -Θ(P ′ , Q r ) 2 F ≤ ⎛ ⎜ ⎝ 1 - 1 3 ⋅ 10 5 log 4 nd δζ- ⎞ ⎟ ⎠ Θ(P , Q r ) -Θ(P , Q r ) 2 F .

D.5. Analysis of BlockSort

Next, we combine the results of the previous sections to control the error of BlockSort. We are given a collection Y of 6τ ∞ samples of the model ( 1) and a valid hierarchical sorting tree T of depth t that we consider as fixed. Then, we take a leaf G of T with maximal depth and we consider (O, P, I) = BlockSort(Y, T , G) the trisection of G, as well as P ⊇ P the more conservative intermediary set. In this section, we provide a high-probability control of P . For any height h ∈ H and scale r ∈ R, recall that Q * cp is the subset(defined in Appendix D.2) of block of questions at scale r such that the mean m(P ) increases by at least h 2. Also recall the superset Q * cp ⊃ Q * cp . At a high level, the next proposition states that, after τ ∞ iterations of the DoubleTrisection -Local routines at all scales r ∈ R and all heights h ∈ H, the size M (P )-M (P ) 2 F is quite small. This is mainly due to the fact that, by Lemma D.3, at each step τ , there exists some (r, h) ∈ R × H such that the norm of the thresholded aggregated matrix

Θ(P τ , Q * cp ) -Θ(P τ , Q * cp ) √
rh is of the same order as M (P ) -M (P ) 2 F . By Lemma D.1, the estimated blocks Q * cp contain Q * cp with high probability. Hence, unless the norm of the thresholded aggregated matrix is small, we derive from corollary D.6 that the norm of this aggregated matrix has contracted at step τ + 1. Hence, after τ ∞ steps, one could expect that the norm of M (P ) -M (P ) 2 F is small. In fact, both the statement and the proof of this proposition are slightly more involved because we need to keep track of the scales and heights of interest. Define the function Ψ(p, r, h, q) by

Ψ(p, r, h, q) = hp √ rq ζ ∧ ( √ pq) + p . (64)
Proposition D.7. With probability higher than 1 -5τ ∞ δ, there exists a subset P † such that

P ⊆ P †
⊆ G and the following property holds. For some r † ∈ R and some h † ∈ H, upon writing

Q † cp = Q * cp (P † , h † , r † ) and Q † cp = Q * cp (P † , h † , r † ), we have simultaneously Θ(P † , Q † cp ) -Θ(P † , Q † cp ) √ r † h † 2 F ≤ 4 ⋅ 10 5 ζ 2 log 3 6nd δζ - Ψ( P † , r † , h † , Q † cp ) ; (65)
M (P † ) -M (P † ) 2 F ≤ 16ζ 2 + 96 R H [Θ(P † , Q † cp ) -Θ(P † , Q † cp )] √ r † h † 2 F . (66)
In other words, there exists a superset P † of P such that, for a suitable height and scale, at the high-variation regions, both the original matrix M (P †

) and the thresholded aggregated matrix are controlled at the level Ψ( P † , r † , h † , Q † cp ). The virtue of the above result is that it easily adapts to the block sorting variant with memory. Unfortunately, the rate

Ψ( P † , r † , h † , Q † cp
) is a bit difficult to handle. In the next corollary, we replace it by a simpler but cruder bound that only depends on G , h † and d.

Corollary D.8. Under the same event of probability higher than 1 -5τ ∞ δ as in the previous proposition, the set P † , the scale r † , and the height h † also satisfy

Θ(P † , Q † cp ) -Θ(P † , Q † cp ) √ r † h † 2 F ≲ ζ 2 log 3.5 6nd δζ - ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ h † G √ d ζ ∧ G d ∧ G h † + G ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ , ( 67
)
where we recall that G is the initial group.

D.6. Analysis of the complete procedure TreeSort

We are now equipped to prove Proposition B.5.

Proof of Proposition B.5. Let us fix an integer t ≤ t ∞ and let us consider the collection L t of the 2 t groups P that are not sorted with confidence. Let us apply Proposition D.7 to each of these sets P . In view of this proposition, we define (P † , h † , r † ) as well as Q † cp . We also define

s † = l ∶ ∃i, j ∈ P † s.t. Θ i,l (P † , Q † cp ) -Θ j,l (P † , Q † cp ) √ r † h † ≠ 0 .
In a nustshell, s † is the number of columns of the thresholded aggregated matrix which are not equal to zero.

Definition 3. Define the dyadic collection S = {1, 2, 4, . . . , 2 ⌈log 2 (d)⌉ }. For any s ∈ S, r ∈ R, and h ∈ H, we consider the collection P * (h, r, s) ⊂ L t satisfying r † = r, h † = h, and s † ∈ [s, 2s).

The following lemma controls the cardinality of P * (h, r, s). This bound mainly relies on the facts that the matrix M is, up to a row permutation, bi-isotonic and that its entries lie in

[0, 1].
Lemma D.9. Assume that there exists an ordering σ of L t that orders all groups P 's. In other words, for any r ≤ s, any expert i ∈ P σ(r) is below any expert j ∈ P σ(s) . Then, upon this assumption,

P * (h, r, s) ≤ 2d hrs ∧ 2 t ≤ d2 t+1 hrs ,
for any h ∈ H, r ∈ R, and s ∈ S.

In fact, all the collections L t with t = 0, . . . , t ∞ satisfy the assumption in the above under the event ξ defined in Corollary B.4 -see the proof of Proposition B.3.

Putting everything together and summing over the groups P * (h, r, s), we derive from Proposition D.7 and Corollary D.8 that, with probability higher than 1 -5 ⋅ 2 t τ ∞ δ, we have

P ∈Lt M (P ) -M (P ) 2 F ≤ 16ζ 2 L t + 96 R H h,r,s P ∈P * (h,r,s) Θ(P , Q † cp ) -Θ(P , Q † cp ) √ rh 2 F (a)
≲ ζ 2 log 5.5 6nd δζ -h,r,s P ∈P * (h,r,s)

n 2 t ζ 2 s † rh 2 ∧ n 2 t h + n 2 t (b) ≲ ζ 2 log 5.5 6nd δζ -h,r,s ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ nsrh 2 ζ 2 ∧ dn srh 2 + n ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ (c) ≲ ζ 2 log 8.5 6nd δζ - n 2 3 d 1 3 ζ 2 3 + n ,
where in (a), we combined Corollary D.8 with the fact that the size of each group is at most n 2 t , the crude bound

[A] η 2 F ≤ η 2 d 1 d 2 for any d 1 × d 2 matrix and that n 2 t ≥ 1.
In (b), we relied on Lemma D.9, whereas in (c) we used that x ∧ y ≤ x 1 3 y 2 3 .

We have proved the desired n 2 3 d 1 3 ζ 2 3 + n upper bound. The rate nd 1 6 ζ 1 3 is proved using the same scheme except that we apply Corollary D.8 differently in (a). More precisely, we have

P ∈Lt M (P ) -M (P ) 2 F ≤ 16ζ 2 L t + 96 R H h,r,s P ∈P * (h,r,s) Θ(P , Q † cp ) -Θ(P , Q † cp ) √ rh 2 F ≤ ζ 2 log 5.5 6nd δζ -h,r,s P ∈P * (h,r,s) n 2 t ζ h √ d ∧ n 2 t h ∧ n 2 t d + n 2 t ≲ ζ 2 log 5.5 6nd δζ -h r,s P * (h, r, s) n 2 t ζ h √ d ∧ n 2 t h ∧ n 2 t d + n 2 t (a ′ ) ≲ ζ 2 log 5.5 6nd δζ -h ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ nh √ d ζ ∧ n 2 h ∧ √ n2 t d + n ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ (b ′ ) ≲ ζ 2 log 6.5 6nd δζ - nd 1 6 ζ 1 3 ∧ n √ d + n ,
where in (a ′ ), we used that ∑ r,s P * (h, r, s) ≤ 2 t ≤ 2n and in (b') that xy ≤ x 1 3 y 2 3 .

Proof of Lemma D.9. To ease the notation, we write P * = P * (h, r, s) in this proof. Since P * ⊂ L t , we straightforwardly derive that P * ≤ L t ≤ 2 t . Let us introduce the width of a matrix Θ ∈ R [n]×Qr on the set P ⊂ [n] and Q ⊂ Q r :

W ∞,1 (Θ, P, Q) = max i,j∈P l∈Q Θ i,l -Θ j,l .
Consider any set P and the corresponding quantities P † , s † , r † , and h † . By definition of s † ,

we have W ∞,1 (Θ, P † , Q † cp ) ≥ s † √ r † h † .
Recall that the matrix Θ is, up to a permutation of its rows, bi-isotonic. Besides, all the groups P in L t are perfectly ordered by assumption. As a consequence, the width of Θ on [n] is larger or equal to the sum of the width on each set P . Since P * is an ordered sub-partition, it holds that

W ∞,1 (Θ, [n], Q r ) ≥ P ∈P * W ∞,1 (Θ, P † , Q r ) ≥ P ∈P * W ∞,1 (Θ, P † , Q † cp ) ≥ P * s † √ rh ≥ P * s √ rh . (68) By definition of Q r , we have Q r ≤ 2d r. Since the values of Θ lie in [0, √ r], we deduce that W ∞,1 (Θ, [n], Q r ) ≤ 2d √ r.
Together with (68), this yields

P * ≤ 2d rsh ,
which concludes the proof.

y i = m i = 0 if i ≤ 0 and y i = m i = 1 if i > d.
Since the CUSUM statistic is linear, we have the decomposition

Ĉk,r = C * k,r + 1 r k+r-1 k ′ =k e k ′ (P ) - k-1 k ′ =k-r e k ′ (P ) ,
where the latter random variable is centered and ζ( P r 2) -1 2 -subGaussian. By a union bound, we derive that, with probability higher than 1 -δ, we have

max k∈[d] Ĉk,r -C * k,r ≤ ζ 2 ⋅ 2 P r log 2d δ .
Since r is defined in such a way that

ζ 4 P log 2d δ ≤ 1 8 √ rh , we deduce that Dcp (h, r) ⊂ D * cp (h, r). Conversely, if k belongs to D * cp (h, r), we have C * k,8r ≥ h 2. Since m(P ) is an isotonic vector and r ≥ 8r, it follows that C * k,r ≥ C * k,8r ≥ h 2. We deduce that Ĉk,r ≥ h[ 1 2 -1 8 ] ≥ h 4 , which implies that D * cp (h, r) ⊂ Dcp (h, r).
Proof of Lemma D.2. If an index k belongs to D * (h, r), this implies that m k+r -m k-r ≥ h 8, since the vector m is isotonic. Define κ = 1 + ⌈r r⌉. Since m is an isotonic vector, for l ∈ Q * cp (h, r), we deduce that m l+κr -m l-κr ≥ h 8. Consider the regular grid Q κr of width κr and define

Q * (h, r, κ) = {l ∈ Q κr ∶ Q * cp (h, r) ∩ [l, l + κr) ≠ ∅}. Since, for l ∈ Q * (h,
r, κ), we have m l+2κr -m l-2κr ≥ h 8 and since the total variation of m is at most one, this implies

h 8 Q * (h, r, κ) ≤ l∈D(κ,r,h) m l+2κr -m l-2κr ≤ l∈Qκr m l+2κr -m l-2κr ≤ 4 . Since Q * (h, r) ≤ κ Q * (h, r, κ
) and since κ ≤ 2r r, we obtain the desired result.

Proof of Lemma D.3. For any height h ∈ H -recall the definition of the dyadic class H in (36)and any expert i ∈ P , we consider the h-level set M i,.

m, that is

F (i, h) = {k ∈ [d] ∶ M i,k -m k ≥ h} ; F (i, -h) = {k ∈ [d] ∶ M i,k -m k ≤ -h} . ( 69
)
Since F (i, h) and F (i, -h) are subsets of [d], we can decompose them into unions of disjoint intervals. For any positive integer r ∈ R, we write F (i, h, r) as the union of intervals of F (i, h) whose size belongs [2r -1, 4r -1). Finally, we consider the subset F (i, h, r; 2h) ⊂ F (i, h, r) of all intervals of F (i, h, r) that intersect F (i, 2h). In other words, any maximal interval I in F (i, h, r; 2h) is a h-level set whose size belongs to [2r -1, 4r -1) and such that M i,.

m crosses the level 2h in I. We define similarly F (i, h, r) and F (i, h, r; 2h) when h is negative and -h ∈ H. It follows from these definitions that, for any h such that either h ∈ H or -h ∈ H, we have

F (i, 2h) ⊂ ⋃ r∈R F (i, h, r; 2h) . ( 70
)
We define F * (h, r, 2h) as the union of those intervals for i ∈ P .

F * (h, r, 2h) = ⋃ i∈P F (i, h, r; 2h) .
First, we claim that this collection of intervals F * (h, r, 2h) is contained in the significant regions of variation of m. This result heavily relies on the monotonicity assumptions.

Lemma E.1. For any h ∈ H and any r ∈ R.

[

F * (h, r, 2h) ⋃ F * (-h, r, -2h)] ⊂ D * cp (h, r) .
Next, we quantify M (P ) -M (P ) 2 F using regions of large variation of M i,. -m. Lemma E.2. For any P , it holds that

M (P ) -M (P ) 2 F ≤ 16 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ζ 2 + i∈P r∈R,h∈H h 2 ( F (i, h, r; 2h) + F (i, -h, r; -2h) ) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦
.

The last lemma connects these sets F (i, h, r; 2h) to the norm of the thresholded aggregated matrix.

Lemma E.3. For any r ∈ R and h ∈ H, we consider Θ(P , Q * cp (h, r)) the aggregation of M at scale r and at Q * cp (h, r). We have

h 2 i∈P [ F (i, h, r; 2h) + F (i, -h, r; -2h) ] ≤ 3 Θ(P , Q * cp (h, r)) -Θ(P , Q * cp (h, r)) √ rh 2 F .
Combining Lemmas E.2 and E.3, we conclude that

M (P ) -M (P ) 2 F ≤ 16 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ζ 2 + i∈P r∈R, h∈H h 2 ( F (i, h, r; 2h) + F (i, -h, r; -2h) ) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ 16 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ζ 2 + 3 r∈R, h∈H Θ(P , Q * cp (h, r)) -Θ(P , Q * cp (h, r)) √ rh 2 F ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ 16 ζ 2 + 6 R H max r∈R, h∈H Θ(P , Q * cp (h, r)) -Θ(P , Q * cp (h, r)) √ rh 2 F ,
which concludes the proof of Lemma D.3.

Proof of Lemma E.1. Consider any i ∈ P , any height h ∈ H, and any scale r ∈ R. Without loss of generality, we only focus on F (i, h, r; 2h); the case of F (i, -h, r; -2h) being analogous.

Let I be an interval of F (i, h, r; 2h). Fix any question k ∈ I such that M i,k -m k ≥ 2h. Since k ∈ F (i, h, r), it follows that there exists l < 4r such that M i,k+l -m k+l ≤ h. Since both the vectors M i,⋅ and m are isotonic, it follows that m k+l -m k ≥ h. Now consider any k 0 ∈ I. Using again the monotonicity of m, we deduce that,

C * k 0 ,8r ≥ 4rh 8r ≥ 1 2 h ,
and k 0 therefore belongs to D * cp (h, r). We have proved the desired result.

Proof of Lemma E.2. Consider any expert i ∈ P . We decompose the norm of [M i,⋅ -m(P )] using the level sets of this vector. We recall that H is of the form {h min , 2h min , 4h min , . . .} where

h min ∈ [ζ 2 nd, 2ζ 2 nd]. M i,⋅ -m(P ) 2 2 ≤ h∈H d k=1 (M i,k -m k (P )) 2 1{2h ≤ M i,k -m k (P ) < 4h} + 4dh 2 min ≤ 16 h∈H d k=1 h 2 1{2h ≤ M i,k -m k (P ) < 4h} + 16ζ 2 n 2 d ≤ 16 h∈H h 2 F (i, 2h) + 16ζ 2 n 2 d ≤ 16 h∈H r∈R h 2 F (i, h, r; 2h) + 16ζ 2 n 2 d ,
where in the last line, we used (70). Then, we sum over i ∈ P to conclude.

Proof of Lemma E.3. Consider any i ∈ P , any height h ∈ H, and any scale r ∈ R. Without loss of generality, we only consider F (i, h, r; 2h) the case of F (i, -h, r; -2h) being analogous.

Let I be a maximal interval of F (i, h, r; 2h). We deduce from Lemma E.1 that I is included in D * cp (h, r). Let I 0 be the largest sub-interval of I of the form [qr, q ′ r) where q and q ′ ∈ Q r . Since I ≥ 2r -1, it follows that I ≤ 3 I 0 . We write L 0 the subset of columns of the aggregated matrix Θ(P , Q * cp (h, r)) corresponding to I 0 so that L 0 = I 0 r. On each column l of L 0 , we have

Θ i,l (P , Q * cp (h, r)) -θ l (P , Q * cp (h, r)) ≥ √ rh.
Putting everything together, we get

h 2 I ≤ 3h 2 I 0 = 3h 2 r L 0 ≤ 3 l∈L 0 Θ(P , Q * cp (h, r)) i,l -θ(P , Q * cp (h, r)) l √ rh 2 .
Summing over all intervals I and over all experts i ∈ P and also accounting for the F [i, -h, r; -2h] concludes the proof.

E.2. Proof of Proposition D.4

To simplify the notation, we define

Φ l 1 = 2(2ζ √ 2 + β tris ) log 2 P δ = φ l 1 log 2 P δ .
For simplicity, we respectivly write Θ(P ) = Θ(P , Q) and Θ(P

′ ) = Θ(P ′ , Q) in this proof.
Recall that θ(P ) stands the mean row of Θ(P ) whereas θ(P

′

) stands for the mean row of Θ(P ′ ). Invoking lemma C.1 with w = 1 Q and since the matrix Θ is isotonic, we deduce that outside an event of probability smaller than δ, we have

max i,j∈P ′ Θ(P ′ ) i,⋅ -Θ(P ′ ) j,⋅ 1 ≤ Φ l 1 Q . ( 71
)
since the matrix Θ is isotonic. We shall deduce from this inequality the desired bound. We consider two cases depending on the difference between θ(P ) and θ(P

′
) the mean rows in P and P ′ .

Case 1:

P ′ ⋅ θ(P ) -θ(P ′ ) 2 2 > 1 16φ Θ(P ) -Θ(P ) 2 F . Since P ′ ⊂ P , we deduce that Θ(P ) -Θ(P ) 2 F -Θ(P ′ ) -Θ(P ′ ) 2 F ≥ i∈P ′ Θ(P ) i,⋅ -θ(P ) 2 2 -Θ(P ) i,⋅ -θ(P ′ ) 2 2 = P ′ ⋅ θ(P ) -θ(P ′ ) 2 2 ≥ 1 16φ Θ(P ) -Θ(P ) 2 F ,
where we used the condition in the last line. We have proved the desired result. Case 2:

P ′ ⋅ θ(P ) -θ(P ′ ) 2 2 ≤ 1 16φ Θ(P ) -Θ(P ) 2 F .
We start with the decomposition

Θ(P ′ ) -Θ(P ′ ) 2 F ≤ Θ(P ′ ) -Θ(P ) 2 F = Θ(P ) -Θ(P ) 2 F -Θ(P ∖ P ′ ) -Θ(P ) 2 F , (72)
so that we only have to control Θ(P ∖ P ′ ) -Θ(P ) 2 F from below. By definition of the operator [⋅] η , we have

Θ(P ∖ P ′ ) -Θ(P ) 2 F ≥ [Θ(P ∖ P ′ ) -Θ(P )] η 2 F = [Θ(P ) -Θ(P )] η 2 F -[Θ(P ′ ) -Θ(P )] η 2 F .
By assumption, we have [Θ(P ) -Θ(P )] η 2 F ≥ φ -1 Θ(P ) -Θ(P ) 2 F . Hence, as long as we prove that

[Θ(P ′ ) -Θ(P )] η 2 F ≤ (2φ) -1 [Θ(P ) -Θ(P ) 2 F , (73)
we can safely conclude from (72) that

Θ(P ′ ) -Θ(P ′ ) 2 F ≤ (1 -(2φ) -1 ) Θ(P ) -Θ(P ) 2 F .
Thus, we only have to prove (73). Again, by definition of the thresholding operator, we have

[Θ(P ′ ) -Θ(P )] η 2 F = η 2 i∈P ′ , l∈Q 1 Θ i,l -θ(P ) l ≥η ≤ η 2 i∈P ′ , l∈Q 1 Θ i,l -θ(P ′ ) l ≥η 2 + 1 θ(P ′ ) l -θ(P ) l ≥η 2 . ( 74
)
By Markov inequality, the condition that defines Case 2 above implies that

l∈Q 1{ θ(P ′ ) l -θ(P ) l ≥ η 2} ≤ 1 4φ Θ(P ) -Θ(P ) 2 F P ′ η 2 . ( 75
)
From (71) and a convexity argument, we deduce that, for any i ∈ P ′ , Θ(P

′ ) i,⋅ -θ(P ′ ) 1 ≤ Φ l 1 Q .
Then, applying again Markov inequality, we deduce that, for any expert i in P ′ and any η > 0, we have

l∈Q 1{ Θ(P ′ ) i,l -θ(P ′ ) l ≥ η 2} ≤ 2Φ l 1 Q η .
Since we assume that Θ(P ) -

Θ(P ) 2 F ≥ 8φΦ l 1 η P Q ≥ 8φΦ l 1 η P ′ Q , we deduce that l∈Q 1{ Θ(P ′ ) i,l -θ(P ′ ) l ≥ η 2} ≤ 1 4φ Θ(P ) -Θ(P ) 2 F P ′ η 2 . ( 76
)
So that, combining (74), ( 75) and (76), we arrive at

[Θ(P ′ ) -Θ(P )] η 2 F ≤ 1 2φ Θ(P ) -Θ(P ) 2 F .
We have proved (73).

E.3. Proof of Proposition D.5

For simplicity, we write in this proof Θ ∶= Θ( P , Q) and Θ(P ′ ) ∶= Θ(P ′ , Q). Without loss of generality, we assume that the rows of Θ are already ordered according to the oracle order so that Θ is bi-isotonic.

First, the following lemma states that, the first singular value of (Θ -Θ) is, up to polylogarithmic terms, of the same order as its Frobenius norm. This is mainly due to the fact that the entries of Θ lies in [0, √ r] and that Θ is a bi-isotonic matrix.

Lemma E.4. Assume that Θ -Θ F ≥ 2ζ. For any sets P and Q, we have

Θ -Θ 2 op ≥ 1 16 log 2 (nd ζ -) Θ -Θ 2 F . Now, write v = arg max v 2 ≤1 v T (Z (1) -Z ( 1
)
) 2 2 -1 2 v T (Z (1) -Z (1)
-

Z (2) + Z (2)
) 2 2 .

Lemma E.5. Fix any δ ∈ (0, 1). If

Θ -Θ 2 op ≥ 1600ζ 2 Q (5 P + log(6 δ)) + 7 P + 2 log(6 δ) , (77)
then, with probability higher than 1 -δ, we have

vT Θ -Θ 2 2 ≥ 1 2 Θ -Θ 2 op .
In light of Lemma E.4 and Condition (62), the Condition (77) in Lemma E.5 is valid. Consequently, there exists an event of probability higher than 1 -δ such that

vT Θ -Θ 2 2 ≥ 1 32 log 2 (nd ζ -) Θ -Θ 2 F . (78)
Next, we show that a thresholded version of ẑ = (Z (3) -Z

) T v is almost aligned with z * = (Θ -Θ) T v. We define the sets S * ⊂ Q and Ŝ ⊂ Q of blocks of questions by

S * = l ∈ Q ∶ z * l ≥ 3ζ 2 log(2 Q δ) ; Ŝ = l ∈ Q ∶ ẑl ≥ 2ζ 2 log(2 Q δ) .
S * stands for the collection of blocks of questions l such that z * l is large whereas Ŝ is the collection of blocks l with large ẑl . Finally, we consider the vectors w * and ŵ defined as theresholded versions of z * and ẑ respectively, that is w * i = z * i 1 i∈S * and ŵi = ẑi 1 i∈ Ŝ . Note that, up to the sign, ŵ stands for the active coordinates computed in DoubleTrisection -PCA.

We write v for any unit vector in R P . Since the noise matrix N (2) is made of independent ζ-subGaussian random variables, it follows that (v T (N (3) -N

)) l is a ζ-subGaussian random variables. Hence, we deduce that, for any fixed matrix Θ, subsets P and Q, and any unit vector v, we have

P max l∈Q (v T (N (3) -N (3)
)

) l ≤ ζ 2 log(2 Q δ) ≥ 1 -δ .
Observe that ẑ = z * + vT (N (3) -N

). Conditioning on v, we deduce that, on an event of probability higher than 1 -δ, we have

ẑ -z * ∞ ≤ ζ 2 log(2 Q δ) . ( 79
)
Under this event, we have S * ⊂ Ŝ and for l ∈ Ŝ, we have z * l ẑl ∈ [ 1 2, 2]. Next, we shall prove that, under this event, vT (Θ -Θ) ŵ ŵ 2 is large (in absolute value):

vT (Θ -Θ) ŵ = (z * ) T ŵ = l∈ Ŝ z * l ẑl ≥ 2 5 l∈ Ŝ(z * l ) 2 + (ẑ l ) 2 ≥ 2 5 [ w * 2 2 + ŵ 2 2 ] ≥ 4 5 ŵ 2 w * 2 ,
where we used in the first inequality that z * l ẑl ∈ [1 2, 2] and in the second inequality that S * ⊂ Ŝ. Thus, it holds that

vT (Θ -Θ) ŵ ŵ 2 2 ≥ 16 25 w * 2 2 . ( 80
)
It remains to prove that w * 2 is large enough. Writing S * c for the complementary of

S * in Q, it holds that w * 2 2 = z * 2 2 - l∈S * c (z * l ) 2 , ( 81
)
so that we need to upper bound the latter quantity. Write z * S * c = z * -w * . Coming back to the definition of z * ,

l∈S * c (z * l ) 2 2 = l∈S * c [v T (Θ -Θ)] l z * l 2 ≤ Θ -Θ z * S * c 2 2 = i∈ P l∈S * c (Θ i,l -θ l )z * l 2 ≤ 18ζ 2 P 2 log 2 Q δ i∈ P ⎛ ⎜ ⎝ l∈S * c j∈ P Θ i,l -Θ j,l ⎞ ⎟ ⎠ 2 ≤ 18ζ 2 P 2 log 2 Q δ i∈ P ⎛ ⎜ ⎝ j∈ P Θ i,⋅ -Θ j,⋅ 1 ⎞ ⎟ ⎠ 2 ≤ 18ζ 2 φ 2 l 1 log 2 Q δ log 2 P δ P Q ≤ 145ζ 2 log 2 Q P δ P Q 2 ,
where we used the definition of S * in the third line as well as the Condition (71) in the fifth line. We recall that 78), (81), and Condition (62), we deduce that

φ l 1 = 2(2ζ √ 2 + β tris ) ≤ 29ζ is defined in (60). Recall that z * = vT (Θ -Θ). Combining (
w * 2 2 ≥ 1 64 log 2 (nd ζ -) Θ -Θ 2 F ,
which, together with (80), yields

(Θ -Θ) ŵ ŵ 2 2 2 ≥ vT (Θ -Θ) ŵ ŵ 2 2 ≥ 1 100 log 2 (nd ζ -) Θ -Θ 2 F .
Write ŵ(1) and ŵ(2) the positive and negative parts of ŵ respectively so that ŵ = ŵ(1) -ŵ( 2) and ŵ+ = ŵ(1) + ŵ(2) . We obviously have ŵ 2 = ŵ+ 2 . Besides, if the rows of Θ are ordered according to the oracle permutation, then (Θ -Θ) ŵ( 1) and (Θ -Θ) ŵ( 2) are increasing vectors with mean zero. It then follows from Harris' inequality that these two vectors have a nonegative inner product. We have proved that

(Θ -Θ) ŵ+ ŵ+ 2 2 2 ≥ (Θ -Θ) ŵ ŵ 2 2 2 ≥ 1 100 log 2 (nd ζ -) Θ -Θ 2 F . (82)
Equipped with this bound, we are now in position to show that the set P ′ of experts obtained from P when applying the pivoting algorithm with ŵ+ ŵ+ 2 has a much smaller variance. By Lemma C.1, there exists an event of probability higher than 1 -δ such that max i,j∈P

′ ⟨Θ(P ′ ) i,⋅ -Θ(P ′ ) j,⋅ , ŵ+ ŵ+ 2 ⟩ ≤ φ l 1 log 2 P δ
, where we recall that φ l 1 = 2(2ζ √ 2 + β tris ). By convexity, it follows that

(Θ(P ′ ) -Θ(P ′ )) ŵ+ ŵ+ 2 2 2 ≤ φ 2 l 1 log 2 P δ P ′ ≤ φ 2 l 1 log 2 P δ P .
In light of Condition (62), this quantity is small compared to Θ -Θ 2 F :

(Θ(P ′ ) -Θ(P ′ )) ŵ+ ŵ+ 2 2 2 ≤ 1 200 log 2 (nd ζ -) Θ -Θ 2 F , (83)
which together with (82) leads to

(Θ -Θ) ŵ+ ŵ+ 2 2 2 -(Θ(P ′ ) -Θ(P ′ )) ŵ+ ŵ+ 2 2 2 ≥ 1 200 log 2 (nd ζ -) Θ -Θ 2 F . (84)
Since P ′ ⊂ P , we deduce that, for any vector w ′ ∈ R q , we have (Θ -Θ)w ′ 2 2 ≥ (Θ(P ′ ) -Θ(P ′ ))w ′ 2 . It then follows from the Pythagorean theorem that

Θ -Θ 2 F -Θ(P ′ ) -Θ(P ′ ) 2 F ≥ (Θ -Θ) ŵ+ ŵ+ 2 2 2 -(Θ(P ′ ) -Θ(P ′ )) ŵ+ ŵ+ 2 2 2 .
Then, together with (84), we arrive at

Θ(P ′ ) -Θ(P ′ ) 2 F ≤ 1 - 1 200 log 2 (nd ζ) Θ -Θ 2 F .
Proof of Lemma E.4. The proof mainly relies on a discretisation argument. Given any a ∈ R and any matrix U , we define the matrix

[U ] thres a by ([U ] thres a ) i,j = U i,j 1{U i,j ∈ (a, 2a]}. If a is negative, then the interval should be understood as [2a, a). Recall that all the entries of Θ -Θ lie in [- √ r, √ r]
. This allows us to decompose this matrix as follows

r -1 2 (Θ -Θ) = i∈N * [r -1 2 (Θ -Θ)] thres 2 -i + [r -1 2 (Θ -Θ)] thres -2 -i .
All the matrices in this decomposition have disjoint support. For all i > log 2 (nd ζ), all the entries of the discretised matrices in the decomposition are smaller than ζ(nd) -1 . Since P = p ≤ n and Q = q ≤ d r, this implies that

i∈N * ,i>log 2 (nd ζ) [r -1 2 (Θ -Θ)] thres 2 -i + [r -1 2 (Θ -Θ)] thres -2 -i 2 F ≤ ζ 2 nd (nd) 2 r ≤ ζ 2 r .
Coming back to the previous bound, we arrive at

r -1 2 (Θ -Θ) 2 F ≤ i∈N * ,i≤log 2 (nd ζ) [r -1 2 (Θ -Θ)] thres 2 -i 2 F + [r -1 2 (Θ -Θ)] thres -2 -i 2 F + ζ 2 r .
As we assume that Θ -

Θ F ≥ 2ζ ≥ 2ζ √ r, r -1 2 (Θ -Θ) 2 F ≤ 4 3 i∈N * ,i≤log 2 (nd ζ) [r -1 2 (Θ -Θ)] thres 2 -i 2 F + [r -1 2 (Θ -Θ)] thres -2 -i 2 F .
Hence, there exists an integer

i 0 ∈ [1, log 2 (nd ζ)] such that 3 Θ -Θ 2 F 8r log 2 (nd ζ) ≤ [r -1 2 (Θ -Θ)] thres 2 -i 0 2 F ∨ [r -1 2 (Θ -Θ)] thres -2 -i 0 2 F .
Assume w.l.o.g. that, for this i 0 ≤ log 2 (nd ζ), we have

3 Θ -Θ 2 F 8r log 2 (nd ζ) ≤ [r -1 2 (Θ -Θ)] thres 2 -i 0 2 F .
Now, we define a different discretised version. For a matrix U and some a ∈ R + , let [U ] a be defined by ([U ] a ) ij = (a1{U i,j ≥ a}) i,j . We readily deduce that

Θ -Θ 2 F ≤ 32 3 r log 2 (nd ζ) [r -1 2 (Θ -Θ)] 2 -i 0 2 F . (85)
The entries of the matrix [r -1 2 (Θ -Θ)] 2 -i 0 lie in {0, 2 -i 0 }. Up to a permutation of the rows of Θ, we can assume that each column of [r -1 2 (Θ -Θ)] 2 -i 0 is isotonic. One can easily check that a matrix that only takes two values and such that each column is isotonic can be transformed into a bi-isotonic matrix by applying a suitable permutation π 0 to its columns. We denote B the corresponding permuted matrix. Recall that we denote p and q the dimensions of B. Then, define the function φ ∶ [p] → {0, . . . , q} such that φ(i) is the number of non-zero entries in the (p -i + 1)-th row of B. Since B is bi-isotonic, the function φ is non-increasing. Besides, we have

p i=1 φ(i) = 2 i 0 i,j B i,j = 2 2i 0 i,j B 2 i,j = 2 2i 0 B 2 F .
Lemma E.6. Let d 1 and d 2 be two positive integers and consider a non-increasing function

f ∶ [d 1 ] → R + . Then, there exists m ∈ [d 1 ] such that ∑ d 1 i=1 f (i) ≤ log(ed 1 )mf (m).
Applying this lemma to φ, we deduce that, for some m ∈ [p], we have

2 2i 0 B 2 F ≤ log(ep)mφ(m) . (86)
Since φ(m) is the number of non-zero entries on the p+1-m-th row of B, since B is bi-isotonic and since B only takes two values, this implies that B contains in the lower right a rectangle of size m × φ(m) with value 2 -i 0 . Define the vector u ∈ R p such that u i = m -1 2 if i ≥ pm + 1 and u i = 0, otherwise. Define also the vector v ∈ R q such v j = 1 φ(m) if j ≥ qφ(m) + 1, and v j = 0 otherwise. It follows from these definitions that u T Bv = 2 -i 0 mφ(m). Recall that [r -1 2 (Θ -Θ)] 2 -i 0 corresponds to a row and column permutation of B. Hence, there exist two permutations π 1 and π 2 such that

u T π 1 [r -1 2 (Θ -Θ)] 2 -i 0 v π 2 = u T Bv .
By construction, the entries of Θ -Θ are higher than 2 -i 0 for all entries such that (u π 1 ) i ≠ 0 and (v π 1 ) j ≠ 0. We deduce that

Θ -Θ op ≥ √ ru T π 1 r -1 2 (Θ -Θ)v = √ r2 -i 0 mφ(m) ≥ √ r log(ep) B F .
Finally, we come back to (85) to conclude that Θ -

Θ op ≥ [32 log(ep) log 2 (nd ζ) 3] -1 2 Θ - Θ F , where we recall that ζ -< ζ.
Proof of Lemma E.6. Define a = sup d 1 m=1 mf (m). As a consequence, we have f (m) ≤ a m. This implies that

d 1 i=1 f (i) ≤ d 1 i=1 a i
≤ a log(ed 1 ) .

We have proved that log(ed 1 )

sup d 1 m=1 mf (m) ≥ ∑ d 1 i=1 f (i).
Proof of Lemma E.5. We start with the two following lemmas. For short, we write p = P and q = Q in this proof.

Lemma E.7. Let N ′ denote a random d 1 ×d 2 matrix whose entries follow independent, centered and ζ-subGaussian distributions. Let Ω ⊂ R d 2 be a subspace of dimension d ′ 2 . With probability larger than 1 -δ, one has

sup u∈R d 1 , v∈Ω∶ u 2 ≤1, v 2 ≤1 u T (N ′ -N ′ )v ≤ 10ζ d 1 + d ′ 2 + log(2 δ) ,
where

N ′ = d -1 1 1 d 1 1 T d 1 N
′ is made of the mean row of N ′ . Lemma E.8. Let N ′ be a random d 1 × d 2 matrix whose entries follow independent, centered and ζ-subGaussian distributions. It holds with probability larger than 1 -δ that

sup u∈R d 1 ∶ u 2 ≤1 u T (N ′ -N ′ ) 2 2 -E u T (N ′ -N ′ ) 2 2 ≤ 64ζ 2 d 2 (5d 1 + log(2 δ) + (5d 1 + log(2 δ)) .
We have Z (1) -Z

= Θ -Θ + N (1) -N (1) ,

so that, for any v ∈ R p , v T (Z (1) -Z (1)
)

2 2 = v T (Θ -Θ) 2 2 + v T N (1) -v T N (1) 2 2 + 2⟨v T N (1) -v T N (1) , v T (Θ -Θ)⟩ , which, in turn, implies that v T (Z (1) -Z (1)
) 2 2 -v T (Θ -Θ) 2 2 -E v T N (1) -v T N (1) 2 2 ≤ (87) v T N (1) -v T N (1) 2 2 -E v T N (1) -v T N (1) 2 2 + 2 ⟨v T N (1) -v T N (1) , v T (Θ -Θ)⟩ .
Write W ⊂ R q for the image of (Θ -Θ) T . Then, we apply Lemma E.7 to derive that sup

v∈R p ∶ v 2 ≤1
⟨v T (N (1) -N

)

, v T (Θ -Θ)⟩ ≤ Θ -Θ op sup v∈R p ∶ v 2 ≤1, u∈W ∶ u 2 ≤1 v T (N (1) -N (1)
)u

≤ 10ζ Θ -Θ op 2p + log(6 δ) , (88)
with probability higher than 1 -δ 3 since the dimension of W is no larger than p. We deduce from Lemma E.8 that, with probability higher than 1 -δ 3, we have

sup v∈R p ∶ v 2 ≤1 v T (N (1) -N (1)
) 2 2 -E v T (N (1) -N (1)
) 2 2 ≤ 64ζ 2 q(5p + log(6 δ)) + 5p + log(6 δ) .

Together with (87) and (88), we have that with probability larger than 1 -2δ 3, sup

v∈R p ∶ v 2 ≤1 v T (Z (1) -Z (1)
) 2 2 -v T (Θ -Θ) 2 2 -E v T N (1) -v T N (1) 2 2
≤ 10ζ Θ -Θ op 2p + log(6 δ) + 64ζ 2 q(5p + log(6 δ)) + (5p + log(6 δ)) .

In the same way, we have that, with probability larger than 1 -δ 3, sup

v∈R p ∶ v 2 ≤1 1 2 v T (Z (1) -Z (1)
-

Z (2) + Z (2)
) 2 2 -E v T (N (1) -N (1)
)

2 2 = 1 2 sup v∈R p ∶ v 2 ≤1 v T (Z (1) -Z (1)
-

Z (2) + Z (2) ) 2 2 -E v T (Z (1) -Z (1)
-

Z (2) + Z (2)
) 2 2 ≤ 128ζ 2 q(5p + log(6 δ) + (5p + log(6 δ)) .

Putting everything together we conclude that, on an event of probability higher than 1 -3δ, we have simultaneously for all v ∈ R p with v 2 ≤ 1 that

v T (Z (1) -Z (1)
) 2 2 -v T (Θ -Θ) 2 2 - 1 2 v T (Z (1) -Z (1)
-

Z (2) + Z (2) ) 2 2
≤ 10ζ Θ -Θ op 2p + log(6 δ) + 192ζ 2 q(3p + log(6 δ)) + (3p + log(6 δ)) .

Since Θ -Θ 2 op ≥ 1600ζ 2 [ q(5p + log(6 δ)) + 7p + 2 log(6 δ)], we deduce that, on the same event, we have

sup v∈R p ∶ v 2 ≤1 v T (Z (1) -Z (1)
) 2 2 -v T (Θ-Θ) 2 2 - 1 2 v T (Z (1) -Z (1)
-Z (2) +Z

) 2 2 ≤ 1 4 Θ-Θ 2 op . Writing ψ(v) = v T (Z (1) -Z (2)
) 2 2 -1 2 v T (Z (1) -Z (1)
-

Z (2) + Z (2)
) 2 2 , we deduce that, for

v such that v T (Θ -Θ) 2 2 = Θ -Θ 2 op , we have Ψ(v) ≥ 3 4 Θ -Θ 2 op , whereas, for v such that v T (Θ -Θ) 2 2 < 1 2 Θ -Θ 2 op , we have Ψ(v) < 3 4 Θ -Θ 2 op . We conclude that v satisfies vT (Θ -Θ) 2 2 > 1 2 Θ -Θ 2 op .
Proof of Lemma E.7. We start with a classical result. Variants of it can be found in random matrix textbooks (see e.g [START_REF] Vershynin | High-dimensional probability: An introduction with applications in data science[END_REF]). Still, we provide a simple dedicated proof below for the sake of completeness.

Lemma E.9. Let N ′ be a d 1 × d 2 matrix whose entries follow independent, centered, and ζ-subGaussian distributions. Consider any vector subspace Ω ⊂ R d 2 with dimension d ′ 2 . With probability higher than 1 -δ, one has

sup u∈R d 1 , v∈Ω∶ u 2 ≤1, v 2 ≤1 u T N ′ v ≤ 5ζ d 1 + d ′ 2 + log(1 δ) .
We have the following decomposition

sup u∈R d 1 ,v∈Ω u 2 ≤1, v 2 ≤1 u T (N ′ -N ′ )v ≤ sup u∈R d 1 ,v∈Ω u 2 ≤1, v 2 ≤1 u T N ′ v + sup u∈R d 1 ,v∈Ω u 2 ≤1, v 2 ≤1 u T N ′ v .
The first expression in the right-hand side is handled with Lemma E.9. Regarding the second one, we observe that N ′ v is a constant vector. As a consequence,

sup u∈R d 1 ,v∈Ω u 2 ≤1, v 2 ≤1 u T N ′ v ≤ d 1 sup v∈Ω n ′ v 2 ,
where n ′ is a ζ √ d 1 -subGaussian random vector. Then, we control this expression applying Lemma E.9 to a 1 × d ′ 2 matrix. All in all, we have proved that, with probability higher than 1 -δ, we have

sup u∈R d 1 ,v∈Ω u 2 ≤1, v 2 ≤1 u T (N ′ -N ′ )v ≤ 10ζ (d 1 + d ′ 2 ) + log 2 δ .
Proof of Lemma E.9. Let U d ( ) denote the -covering number of the d-dimensional unit ball and let U d ( ) denote a corresponding minimal covering set. For

Ω a d ′ 2 -dimensional subspace of R d ′ 2
, we also write with a slight abuse of notation U d ′ 2 ( ) for a corresponding minimal covering set of its unit ball. Consider any

d 1 × d 2 matrix W . Write w * = sup u∶ u 2 ≤1 sup v∈Ω, v 2 ≤1 u T W v and w = sup u∈U d 1 (1 4) sup v∈U d ′ 2 (1 4) u T W v . Given u ∈ R d 1 , let π(u) denote any closest point of u in U d 1 (1 4). Similarly, for v ∈ Ω, π ′ (v) stands for a closest point of v in U d ′ 2 ( 1 4)
. By triangular inequality, we have

w * ≤ w + sup u∶ u 2 ≤1 sup v∈Ω, v 2 ≤1 u T W v -π(u)W π ′ (v) ≤ w + sup u∶ u 2 ≤1 sup v∈Ω, v 2 ≤1 (u T -π(u) T )W v + π(u) T W (v -π ′ (v)) ≤ w + w * 2 .
We have proven that

sup u∶ u 2 ≤1 sup v∈Ω, v 2 ≤1 u T W v ≤ 2 sup u∈U d 1 (1 4) sup v∈U d ′ 2 (1 4) u T W v . (89)
Since log(U d ( )) ≤ d log(3 ) (see e.g. [START_REF] Wu | Lecture notes on information-theoretic methods for high-dimensional statistics[END_REF]), we deduce from triangular inequality that, with probability higher than 1 -δ, we have

sup u∈R d 1 , v∈Ω∶ u 2 ≤1, v 2 ≤1 u T N ′ v ≤ 2ζ 2(d 1 + d ′ 2 ) log(12) + 2 log(1 δ) .
Proof of Lemma E.8. Relying on (89

) with W = (N ′ -N ′ )(N ′ -N ′ ) T -E (N ′ -N ′ )(N ′ -N ′ ) T , we derive that sup u∶ u 2 ≤1 u T (N ′ -N ′ ) 2 2 -E u T (N ′ -N ′ ) 2
2 is less than or equal to 2 sup

u∈U d 1 (1 4) sup v∈U d 1 (1 4) u T (N ′ -N ′ )(N ′ -N ′ ) T v -E u T (N ′ -N ′ )(N ′ -N ′ ) T v
As a consequence, it amounts to simultaneously control U d 1 (1 4) 2 quadratic forms of sub-Gaussian random variables. For this purpose, we use the Hanson-Wright inequality [START_REF] Vershynin | High-dimensional probability: An introduction with applications in data science[END_REF]. Below we provide a version of this inequality with explicit numerical constants.

Lemma E.10. Let x be d-dimensional ζ-subGaussian centered random vector with independent components. For any d × d matrix A and any t > 0, we have

P x T Ax -E[x T Ax] ≥ 32ζ 2 A F √ t + A op t ≤ 2e -t
For any fixed u and v, we interpret u

T (N ′ -N ′ )(N ′ -N ′
) T v as a quadratic form of d 1 d 2 independent random variables where the corresponding matrix B of the quadratic form satisfies

B op ≤ 1 and B F ≤ √ d 2 .
Putting everything together we deduce that, with probability higher than 1 -δ, we have

sup u∶ u 2 ≤1 u T (N ′ -N ′ ) 2 2 -E u T (N ′ -N ′ )) 2 2 ≤ 64ζ 2 d 2 (2d 1 log(12) + log(2 δ) + 2d 1 log(12) + log(2 δ) ≤ 64ζ 2 d 2 (5d 1 + log(2 δ) + (5d 1 + log(2 δ)) .
Proof of Lemma E.10. We consider separately the diagonal terms of A and the non-diagonal terms. Write A -for the matrix such that A - ij = A ij 1 i≠j . First, we use Section 2.8 in [22] to handle x T A -x. We know that

P x T A -x ≥ 8ζ 2 A - F √ t + √ 2 A - op t ≤ e -t ,
for any t > 0. Regarding the diagonal part, we know from Rudelson and Vershynin [START_REF] Rudelson | Hanson-wright inequality and sub-gaussian concentration[END_REF] (Step 1 of the main proof) that [START_REF] Vershynin | High-dimensional probability: An introduction with applications in data science[END_REF] for a definition of . ψ 1 ). Then, we are in position to apply Bernstein's inequality [START_REF] Boucheron | Concentration inequalities[END_REF]

x 2 i -E[x 2 i ] ψ 1 ≤ 4ζ 2 (see
(Theorem 2.10) to ∑ i a ii x 2 i with v = (16ζ 2 ) 2 ∑ i a 2
ii and c = 16ζ 2 max i a ii . For any t > 0, we have

P ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ d i=1 a ii (x 2 i -E[x 2 i ]) ≥ 16ζ 2 ⎛ ⎝ 2 i a 2 ii t + max i a ii t ⎞ ⎠ ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ e -t ,
which implies that

P d i=1 a ii (x 2 i -E[x 2 i ]) ≥ 16ζ 2 A F √ 2t + A op t ≤ e -t .
We combine the two deviation inequalities and use A - op ≤ 2 A op to conclude that

P x T Ax -E[x T Ax] ≥ 32ζ 2 A F √ t + A op t ≤ 2e -t . E.4. Proof of Corollary D.6
Let (L cp , U cp ) denote the conservative result of Pivot(Z (1) , 1 Q , γ) and 2) , Z (3) , Z (4) , Z (5) ). Here, (Z (2) , Z (3) , Z (4) ) restricted to the experts in P , whereas Z (5) is restricted to experts in P . Finally, we write P ′ = P ∖ (L pca ∪ U pca ). We first prove the following intermediary result

P = P ∖ (L cp ∪ U cp ). Let (L pca , U pca ) = DoubleTrisection -PCA(Z, γ) with Z = (Z (
Θ(P ′ , Q) -Θ(P ′ , Q) 2 F ≤ ⎛ ⎝ 1 - 1 1920 log 2 ( nd δζ-) ⎞ ⎠ Θ(P , Q) -Θ(P , Q) 2 F . (90)
We consider two cases. First, we assume that η ζ P Q ≤ P Q + P . Then, it follows from Equation (63) that we are in position to apply Proposition D.4 with φ = 120 log 2 ( nd δζ-).

Since P ′ ⊂ P , it follows that Θ(P ′ , Q) -Θ(P ′ , Q) 2 F ≤ Θ( P , Q) -Θ( P , Q) 2
F and (90) follows from Proposition D.4. Now, we assume that

P Q + P ≤ η ζ P Q . If Θ( P , Q) -Θ( P , Q) 2 F ≤ 0.5 Θ(P , Q) - Θ(P , Q) 2
F , then the result obviously holds. Otherwise, it follows from (63) that

Θ( P , Q) -Θ( P , Q) 2 F ≥ 2 ⋅ 10 5 log 3 6nd δζ - η P Q ∧ P Q + P ,
Besides, with probability higher than 1 -δ, Θ( P ) is undistinguishable in l 1 -norm by (71). Hence, we are in position to apply proposition D.5 and it follows that

Θ(P ′ , Q) -Θ(P ′ , Q) 2 F ≤ ⎛ ⎝ 1 - 1 200 log 2 ( nd δζ-) ⎞ ⎠ Θ( P , Q) -Θ( P , Q) 2 F ≤ ⎛ ⎝ 1 - 1 1920 log 2 ( nd δζ-) ⎞ ⎠ Θ(P , Q) -Θ(P , Q) 2 F ,
which is exactly Equation (90).

It remains to conclude from Equation (90). We start from

Θ(P ′ , Q r ) -Θ(P ′ , Q r ) 2 F = Θ(P ′ , Q) -Θ(P ′ , Q) 2 F + Θ(P ′ , Q r ∖ Q) -Θ(P ′ , Q r ∖ Q) 2 F ≤ ⎛ ⎝ 1 - 1 1920 log 2 ( nd δζ-) ⎞ ⎠ Θ(P , Q) -Θ(P , Q) 2 F + Θ(P , Q r ∖ Q) -Θ(P , Q r ∖ Q) 2 F ≤ ⎛ ⎜ ⎝ 1 - 1 3 ⋅ 10 5 log 4 nd δζ- ⎞ ⎟ ⎠ Θ(P , Q r ) -Θ(P , Q r ) 2 F ,
where we used in the last line that

Θ(P , Q) -Θ(P , Q) 2 F ≥ 1 (120 log 2 ( nd δζ-)) Θ(P , Q r ) - Θ(P , Q r ) 2
F .

E.5. Proof of proposition D.7

For all τ = 0, . . . , τ ∞ , let (O τ , P τ , I τ ) be the sets defined in BlockSort. Let also

(h †τ , r †τ ) = arg max h,r Θ(P τ , Q * cp ) -Θ(P τ , Q * cp ) √ rh 2 F
, where we recall that Q * cp depends on h and r. For simplicity, we write

Q †τ cp = Q * cp (h †τ , r †τ ).
Equipped with this notation, we readily deduce from Lemma D.3 that

M (P τ ) -M (P τ ) 2 F ≤ 16ζ 2 + 96 R H Θ(P τ , Q †τ cp ) -Θ(P, Q †τ cp ) √ r †τ h †τ 2 F . (91)
If, for some τ < τ ∞ , we have

Θ(P τ , Q †τ cp ) -Θ(P τ , Q †τ cp ) √ r †τ h †τ 2 F ≤ 4 ⋅ 10 5 ζ 2 log 3 6nd δζ - Ψ( P τ , r †τ , h †τ , Q †τ cp ) , (92)
then we can fix, for any such τ , P † = P τ , h † = h †τ , Q † = Q †τ cp , and r † = r †τ so that both the properties (65) and (66) hold.

Hence, we assume henceforth that, for all τ , Equation (92) does not hold and we shall arrive at a contradiction. In particular, this implies that

[Θ(P τ , Q †τ cp ) -Θ(P τ , Q †τ cp )] √ r †τ h †τ 2 F ≥ ζ 2 ≥ ζ 2 ( R H ) -1 .
We have 112 R H ≤ 120 log 2 ( nd δζ-) provided that n is a large enough constant. In light of (91), this implies that, for all τ ,

Θ(P τ , Q †τ cp ) -Θ(P τ , Q †τ cp ) √ r †τ h †τ 2 F ≥ 1 120 log 2 ( nd δζ-) M (P τ ) -M (P τ ) 2 F (93) ≥ 1 120 log 2 ( nd δζ-) Θ(P τ , Q r †τ ) -Θ(P τ , Q r †τ ) 2 F , (94)
where we recall that Q r †τ is the collection of all blocks at scale r †τ and we use the Pythagorean equality in the second line. Applying Lemma D.1 at the scale r †τ ∈ R, at the height h †τ ∈ H, and at all steps τ = 0, . . . , τ ∞ -1, we deduce that the event ξ cp ∶= ⋂ τ∞-1 τ =0 ξ cp (P τ , h †τ , r †τ ) holds with probability at least 1 -τ ∞ δ. Under this event, we write Qτ cp for the estimated set defined at step τ and scales (h †τ , r †τ ) in BlockSort. Then, it holds that Q †τ cp ⊂ Qτ cp ⊂ Q †τ cp and we deduce from (94) that

Θ(P τ , Qτ cp ) -Θ(P τ , Qτ cp ) √ r †τ h †τ 2 F ≥ 1 120 log 2 ( nd δτ -) Θ(P τ , Q r †τ ) -Θ(P τ , Q r †τ ) 2 F .
Since (92) is not satisfied, we also have

Θ(P τ , Qτ cp ) -Θ(P τ , Qτ cp ) 2 F ≥ Θ(P τ , Qτ cp ) -Θ(P τ , Qτ cp ) √ r †τ h †τ 2 F ≥ Θ(P τ , Q †τ cp ) -Θ(P τ , Q †τ cp ) √ r †τ h †τ 2 F ≥ 4 ⋅ 10 5 ζ 2 log 3 6nd δζ - Ψ( P τ , r †τ , h †τ , Qτ cp ) , since Qτ cp ⊂ Q †τ cp .
Hence, we are in position to apply Corollary D.6 at all steps τ with r †τ , P τ , Qτ cp , and η = √ r †τ h †τ . There exists an event of probability higher than 1 -4τ ∞ δ such that, at all steps τ , we have

Θ(P τ +1 , Q r †τ ) -Θ(P τ +1 , Q r †τ ) 2 F ≤ ⎛ ⎜ ⎝ 1 - 1 3 ⋅ 10 5 log 4 nd δζ- ⎞ ⎟ ⎠ Θ(P τ ; Q r †τ ) -Θ(P τ ; Q r †τ ) 2 F .
Together with Equation (93), we deduce that

M (P τ ) -M (P τ ) 2 F -M (P τ +1 ) -M (P τ +1 ) 2 F ≥ Θ(P τ , Q r †τ ) -Θ(P τ +1 , Q r †τ ) 2 F -Θ(P τ +1 , Q r †τ ) -Θ(P τ , Q r †τ ) 2 F ≥ 1 4 ⋅ 10 7 log 6 ( nd δζ-) M (P τ ) -M (P τ ) 2 F .
Hence,

M (P τ 0 ) 2 F ≥ M (P τ 0 ) -M (P τ 0 ) 2 F ≥ M (P τ∞ ) -M (P τ∞ ) 2 F ⎛ ⎝ 1 - 1 4 ⋅ 10 7 log 6 ( nd δζ-) ⎞ ⎠ -τ∞
.

Since (92) does not hold at τ = τ ∞ , this implies that the Frobenius norm in the right-hand side of the above inequality is larger than 2ζ 2 and, in light of the definition of τ ∞ = 4⋅10 7 log 7 ( nd δ(ζ-) 2 ), the right-hand side is larger than 2nd. This contradicts the fact that

M (P τ 0 ) 2 F ≤ nd since the entries of M lie in [0, 1].
Proof of Corollary D.8. To ease the notation in this proof, we simply write P for P † , r for r † ,

Q * cp for Q † cp ,
√ rh P Q * cp ζ ∧ P Q * cp + P ≤ G h √ d ζ ∧ ( G d + G ) . (95)
Second, we know from Lemma D.2 that Q * cp ≤ 64 r rh so that

√ rh P Q * cp ζ ∧ P Q * cp + P ≲ P √ rh ζ ∧ ⎛ ⎝ P r rh + P ⎞ ⎠ . ( 96
)
If r > 32ζ 2 log( 2d δ ) 1 P h 2 then, it follows from the definition (53) of r that r = 8r so that the right-hand side of (96) is at most of the order of P h + P . For a smaller r, we know from (53) that r ≲ ζ 2 log( 2d δ ) ( P h 2 ), which in turn implies that

P √ rh ζ ≲ log( 2d δ ) P h .
Hence, we deduce from (96) that

√ rh P Q * cp ζ ∧ P Q * cp + P ≲ log( 2d δ ) ⎛ ⎝ P h + P ⎞ ⎠ .
Together with (95), this leads us to

√ rh P Q * cp ζ ∧ P Q * cp + P ≲ log( 2d δ ) ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ G h √ d ζ ∧ G √ d ∧ G h + G ⎤ ⎥ ⎥ ⎥ ⎥ ⎦
, which, together with (65) concludes the proof.

Appendix F: Proof of Proposition B.6

In this section, we prove Proposition B.6 which states a tighter bound than Proposition B.5 on TreeSort when we use the variant DimensionReduction -WM to compute πW M . Recall that, for any t = 1, . . . , t ∞ , T t stands for the hierarchical sorting tree built by TreeSort at the beginning of step t. Thus, T t has depth t.

The main difference with the analysis of Proposition B.5 lies in the analysis of the algorithm DimensionReduction -WM, which is the purpose of the next subsection. Then, we combine it with the general scheme of the proof of Proposition B.6 to get the desired bound.

F.1. Analysis of DimensionReduction -WM

The key idea of DimensionReduction -WM is to examine the high-variation regions of the observations not only in a set of experts P but also in the neighboring sets of experts. For this reason, we remind the reader of the notation of DimensionReduction -WM. Through this subsection, we fix the step t ≥ 0 of TreeSort. For simplicity, we write T ∶= T t . Recall that L (0,1) (T ) stands for the set of leaves of T of type 0 or 1. By definition, those leaves are all at depth t. Let us focus on a specific leaf G ∈ L (0,1) (T ), and we consider a subset P of G.

Finally, we recall that we consider an ordering of the leaves L (0,1) (T ) at depth t and centered on G as:

(G (a)
) a∈Z = Order(T , G) , where G (0) = G.

Also, we fix any h ∈ H and r ∈ R. As in DimensionReduction -WM, define

r 0 = 2 9 log(4d R δ) ζ 2 P h 2 and r = 4(⌈r 0 ⌉ dya ∨ r) , (97)
where ⌈r 0 ⌉ dya = 2 ⌈log 2 (r 0 )⌉ is the smaller power of 2 which is larger than r 0 . Up to numerical constants, r is defined as for the original procedure DimensionReduction. If r ≥ r 0 , then we can simply rely on CUSUM statistics at the scale 8r and on the set P to detect high variation regions in P . If h (or P ) is so small that r 0 > r, we applied the CUSUM statistic at a larger scale r in DimensionReduction. In this version, we compute the CUSUM statistics at a scale smaller than r to the price of considering more experts than those in P . If r < ⌈r 0 ⌉ dya , let us consider any r cp ∈ [8r, 2r] ∩ R. We respectively define

a + W M ∶= a + W M (T , G, h, r cp ) = min a ∶ G (1) + ⋅ ⋅ ⋅ + G (a) ≥ 2 11 log(4d R δ) ζ 2 r cp h 2 ; a - W M ∶= a - W M (T , G, h, r cp ) = min a ∶ G (-1) + ⋅ ⋅ ⋅ + G (-a) ≥ 2 11 log(4d R δ) ζ 2 r cp h 2 ,
as the minimum number of groups above and below G in such a way that there are enough experts to detect a h-variation in the mean at the scale r cp . Then, V + rcp and V - rcp stand for the collection of experts in the corresponding groups:

V + rcp ∶= V + rcp (T , P , h) = a + W M ⋃ a=1 G (a) and V - rcp ∶= V - rcp (T , P , h) = -1 ⋃ a=-a - W M G (a) , (98)
Finally, we define

V rcp ∶= V rcp (T , P , h) = V + rcp ∪ V - rcp if r cp ≤ r P if r cp = 2r (99)
which exactly corresponds to the definition at Line 4 and Line 6 of DimensionReduction -WM.

For any k ∈ [d], we recall here the definition of the statistic ∆(ext) k,rcp its deterministic counterpart:

∆(ext) k,rcp = 1 2r cp k+rcp-1 k ′ =k-rcp y k ′ (V + rcp )-y k ′ (V - rcp ) and ∆ * (ext) k,rcp = 1 2r cp k+rcp-1 k ′ =k-rcp m k ′ (V + rcp )-m k ′ (V - rcp ) .
In the notation of ∆(ext) k,rcp , we remove the dependency on V + rcp and V - rcp to simplify the notation. Here, ∆(ext) k,rcp stands for the width between the empirical means of the groups above P and below P . Recall also the definition of the statistic Ĉ(ext) k,2rcp and introduce its deterministic counterpart:

Ĉ(ext) k,2rcp = 1 2r cp ⎛ ⎝ k+2rcp-1 k ′ =k y k ′ (V 2rcp ) - k-1 k ′ =k-2rcp y k ′ (V 2rcp ) ⎞ ⎠ C * (ext) k,2rcp = 1 2r cp ⎛ ⎝ k+2rcp-1 k ′ =k m k ′ (V 2rcp ) - k-1 k ′ =k-2rcp m k ′ (V 2rcp ) ⎞ ⎠ .
Here, Ĉ(ext) k,2rcp stands for the mean CUSUM statistic over the experts in V 2rcp . Consider any r cp ∈ [4r, r] ∩ R. Then, as in the algorithm DimensionReduction -WM, we define the collection of positions where both the width and the CUSUM statistic are large:

DW M ∶= DW M (T , P , h, r, r cp ) = k ∶ ∆(ext) k,rcp ≥ h 16 and Ĉ(ext) k,2rcp ≥ h 16 .
See Figure 5 Then, we consider the collections of blocks QW M (T , P , h, r, r cp ), Q * W M (T , P , h, r, r cp ), and QW M (T , P , h, r, r cp ) of size r. With our notation, this means that

Q * W M (T , P , h, r, r cp ) = Encode -Set(D * W M , r), Q * W M (T , P , h, r, r cp ) = Encode -Set(D * W M , r)
, and QW M (T , P , h, r, r cp ) = Encode -Set( DW M , r). Finally, we consider the unions over all possible r cp ∈ R with 4r ≤ r cp ≤ r:

QW M ∶= QW M (T , P , h, r) = r ⋃ rcp=4r QW M (T , P , h, r, r cp ) ; Q * W M ∶= Q * W M (T , P , h, r) = r ⋃ rcp=4r Q * W M (T , P , h, r, r cp ) ; Q * W M ∶= Q * W M (T , P , h, r) = r ⋃ rcp=4r Q * W M (T , P , h, r, r cp ) .
The following lemma states that, with high probability, QW M is sandwiched between

Q * W M
and Q * W M , so that, on the corresponding event, it is sufficient to study these two quantities. Lemma F.1. Consider any valid hierarchical sorting tree T , any subset P of a leaf G of T , any h ∈ H, and any r ∈ R. With probability at least 1 -δ, it holds that

Q * W M ⊂ QW M ⊂ Q * W M . (100)
Next, we show that the aggregation of M (P ) at Q * W M captures most of the variance of M (P ).

Lemma F.2. Assume that T is a valid hierarchical sorting tree. Then, there exist h ∈ H and r ∈ R such that

M (P ) -M (P ) 2 F ≤ 16ζ 2 + 96 R H Θ(P , Q * W M ) -Θ(P , Q * W M ) √ rh 2 F . (101)
Recall that T t∞ (and in particular also T = T t ) is a valid hierarchical sorting tree under the event ξ of high probability defined in Corollary B.4. This lemma is the counterpart of Lemma D.3 for the oblivious DimensionReduction algorithm. Proposition F.3. Assume that T t is a valid hierarchical sorting tree. Consider a leaf G of T of type 0 or 1 at depth t. With probability higher than 1 -5τ ∞ δ, there exists a subset P † such that P ⊆ P † ⊆ G and the following property holds. For some r † cp ≥ r † ∈ R and some h † ∈ H, upon writing

F.2. Analysis of the variant

Q † W M = Q * W M and Q † W M = Q * W M
, we have simultaneously

Θ(P † , Q † W M ) -Θ(P † , Q † W M ) √ r † h † 2 F ≤ 4 ⋅ 10 5 ζ 2 log 3 6nd δζ - Ψ( P † , r † , h † , Q † W M ) ; (102) M (P † ) -M (P † ) 2 F ≤ 16ζ 2 + 96 R H [Θ(P † , Q † W M ) -Θ(P † , Q † W M )] √ r † h † 2 F . (103) Since M (P ) -M (P ) 2 F ≤ M (P † ) -M (P †
) 2 F , the above proposition controls M (P ) - individually. Still, we are able to show that among all groups G ∈ L (0,1) (T t ) that are refined in the t-th iteration of TreeSort, many of them will correspond to small Q † W M . To formalize this argument, we need to be careful about the dependencies of the quantities under consideration.

M (P ) 2 F in terms of Ψ( P † , r † , h † , Q † W M ).
We start from the ordered collection L (0,1) (T t ) of v ≤ 2 t leaves of types 0 or 1. We write G 1 , . . . , G v for these groups and we are given a collection P 1 ,. . . , P v of subgroups such that P i ⊂ G i for i = 1, . . . , v. Later, we will specify P v = P † v , but those sets can be considered arbitrarily.

For a specific group

P v ⊂ G v , we write Q † W M (P v , h, r) instead of Q † W M
to emphasize its dependency on P v , r and h. Given a positive integer p > 0, we define P * (p) = {P v ∶ P v ∈ [p, 2p)} the collection of groups P v of size in [p, 2p). Lemma F.4. Assume that T t is a valid hierarchical sorting tree. For any h ∈ H, r ∈ R, any integer p, any sequence P v of subsets of G v , it holds that

P ∈P * (p) Q * W M (P , h, r) ≲ log(d) ⎡ ⎢ ⎢ ⎢ ⎣ nd(r 0 ∨ r) rh √ p ∧ d(r 0 ∨ r) r 2 h ∧ nd pr ∧ n(r 0 ∨ r) prh ⎤ ⎥ ⎥ ⎥ ⎦ . ( 104
)
We are now equipped to prove Proposition B.6.

Proof of Proposition B.6. We work under the event ξ (Corollary B.4) ensuring T t∞ and in particular T t is a valid hierarchical sorting tree. For each group G s ∈ L (0,1) (T t ) we apply Proposition F. Then, we derive from Proposition F.3 that, on an additional event of probability higher than 1 -5 ⋅ 2 t τ ∞ δ, we have

P ∈Lt M (P ) -M (P ) 2 F ≤ 16ζ 2 L t + 96 R H p,h,r P † ∈P * (p,h,r) Θ(P † , Q † W M ) -Θ(P † , Q † W M ) √ rh 2 F (a) ≲ ζ 2 log 5 6nd δζ -p,h,r P † ∈P * (p,h,r) ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ [ h 2 pr ζ 2 ∧ 1]p Q * W M (P † , h, r) + p ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ (b) ≲ ζ 2 log 5.5 6nd δζ -p,h,r ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ r r ∨ r 0 p P * (p, h, r) P † ∈P * (p,h,r) Q * W M (P † , h, r) + p P * (p, h, r) ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ (c) ≲ ζ 2 log 5.5 6nd δζ -p,h,r ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ nr r ∨ r 0 P † ∈P * (p,h,r) Q * W M (P † , h, r) + n ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ (d) ≲ ζ 2 log 6 6nd δζ -p,h,r ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ⎛ ⎝ n 3 4 d 1 4 1 (r 0 ∨ r)ph 2 1 4 ∧ n d p(r 0 ∨ r) ∧ n √ ph ∧ nd rh ⎞ ⎠ + n ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≲ ζ 2 log 7 6nd δζ -p,h n 3 4 d 1 4 ζ 1 2 ∧ n √ d ∧ nh ζ √ d ∧ n √ h ∧ √ nd √ h + n (e) ≲ ζ 2 log 7 6nd δζ -p,h n 3 4 d 1 4 ζ 1 2 ∧ n √ d ∧ n 2 3 √ d ζ 1 3 ∧ nd 1 6 ζ 1 3 + n ≲ ζ 2 log 9 6nd δζ - n 3 4 d 1 4 ζ 1 2 ∧ n √ d ∧ n 2 3 √ d ζ 1 3 ∧ nd 1 6 ζ 1 3 + n ,
where we applied Proposition F.3 in (a), Jensen inequality and the definition of r 0 in (b), as well as the bound P * (p, h, r) ≤ n p in (c), Lemma F.4 in (d), and x ∧ y ≤ x 1 3 y 2 3 in (e).

F.4. Remaining proofs

Proof of lemma F.1. It is sufficient to prove that with high probability, D * W M (T , P, h, r cp ) ⊂ DW M (T , P, h, r cp ) ⊂ D * W M (T , P, h, r cp ) for all r cp ∈ [4r, r] ∩ R. Recall that we use the convention that

y i = m i = 0 if i ≤ 0 and y i = m i = 1 if i > d.
Since the CUSUM and the envelope statistics are linear, we have the decompositions

Ĉ(ext) k,2rcp (V 2rcp ) = C * (ext) k,2rcp (V 2rcp ) + 1 2r cp ⎛ ⎝ k+2rcp-1 k ′ =k e k ′ (V 2rcp ) - k-1 k ′ =k-2rcp e k ′ (V 2rcp ) ⎞ ⎠ ∆(ext) k,rcp (V + rcp , V - rcp ) = ∆ * (ext) k,rcp (V + rcp , V - rcp ) + 1 2r cp k+rcp-1 k ′ =k-rcp e k ′ (V + rcp ) -e k ′ (V - rcp ) ,
where the two latter random variables are centered and respectively ζ(r cp V 2rcp ) -1 2 -subGaussian and ζ[r cp ( V + rcp ∧ V - rcp )] -1 2 -subGaussian. By a union bound, we deduce that, with probability higher than 1 -δ, we have simultaneously

max rcp∈[4r,r]∩R max k∈[d] Ĉ * (ext) k,2rcp -C * (ext) k,2rcp ≤ ζ 2 r cp V 2rcp log 4d R δ ; (105) max rcp∈[4r,r]∩R max k∈[d] ∆ * (ext) k,rcp -∆ * (ext) k,rcp ≤ ζ 2 ( V + rcp ∧ V - rcp )r cp log 4d R δ . (106)
To conclude, it suffices to check that V + rcp , V - rcp , and V rcp have been chosen large enough so that the right-hand side of the two above equations is at most h 32.

By definition of V + rcp and V - rcp , we know that

V + rcp ∧ V - rcp ≥ 2 11 log(4d R δ) ζ 2 rcph 2 which implies that (106) is at most h 32. If r cp ≤ r 2, then V 2rr cp = V + 2rr cp + V - 2rr cp ≥ 2 11 log(4d R δ) ζ 2 rcph 2
, which implies that (105) is at most h 32. Finally, for r cp = r, we use

r cp ≥ 4r 0 ≥ 2 11 log(4d R δ) ζ 2 P h 2
and that V = P to conclude that (105) is at most h 32.

Proof of Lemma F.2. In the analysis of DimensionReduction, we introduced in (55) the sets D * cp (P, h, r) of questions such that the corresponding CUSUM of the mean expert in P is above h 2 at scale 8r. Recall the set Q * cp ∶= Q * cp (P, h, r) = Encode -Set(D * cp (P, h, r), r). In Lemma D.3, we stated that, for some h ∈ H and r ∈ R, we have (V + r , V - r ) ≥ h 2} for the questions where the population width between V + r and V - r at scale r is at least h 2. Besides, we define Q * env ∶= Q * env (T , P, h, r) = Encode -Set(D * env , r). If l ∈ Q * cp ∖ Q * env , then for any i, j ∈ P , we have

M (P ) -M (P ) 2 F ≤ 16ζ 2 + 96 R H Θ(P, Q * cp ) -Θ(P, Q * cp ) √ rh 2 F . (107)
Θ i,l -Θ j,l ≤ 1 √ r (l+1)r-1 k ′ =lr m k ′ (V + r ) -m k ′ (V - r ) ≤ 2 √ r∆ ( * ext) lr,r < √ rh .
Hence, it follows that

Θ(P, Q * cp ) -Θ(P, Q * cp ) √ rh 2 F = Θ(P, Q * cp ∩ Q * env ) -Θ(P, Q * cp ∩ Q * env ) √ rh 2 F . (108)
In light of ( 107) and ( 108), we only have to prove that, for any fixed T , P , h, and r, we have

D * cp (P, h, r) ∩ D * env (T , P, h, r) ⊂ ⋃ rcp∈[4r,r]∩R D * W M (T , P, h, r, r cp ) . ( 109
)
Since the remainder of the proof heavily relies on the comparisons between CUSUM statistics for different subsets of experts, we respectively write C * (ext) k,r

(V r ) and ∆ * (ext) k,r (V + r , V - r ) instead of C * (ext) k,r and ∆ * (ext) k,r
to better keep track of the dependencies. Fix any question k ∈ D * cp (P, h, r)∩ D * env (T , P, h, r) and define

r min = max{r ′ ∈ R ∶ C * (ext) k,r ′ (V r ′ ) < h 8} ,
with the convention that max(∅) = 1. r min can be interpreted as the largest scale r ′ in R such that the population CUSUM at scale r ′ applied to V r ′ is smaller than h 8. By definition, we have V 2r = P . As a consequence, for any r ′ ≥ 2r, we have

C * (ext) k,r ′ (V r ′ ) = C * k,r ′ (P ) ≥ C * k,2r (P ) ≥ C *
k,8r (P ) ≥ h 8 since k ∈ D * cp (P, h, r) and since r ≥ 8r (see ( 97)). This implies that r min ≤ r. We consider two distinct cases.

Case 1: r min ≤ 4r. Then, we simply choose r cp = 4r. By definition of r min , we have C * (ext) k,2rcp (V 2rcp ) ≥ h 8. Since k ∈ D * env (T , P, h, r), we can lower bound the envelope statistic as

∆ * (ext) k,rcp (V + rcp , V - rcp ) ≥ 1 4 ∆ * (ext) k,r (V + r , V - r ) ≥ h 8 .
We have proved that k ∈ D * W M (T , P, h, r, r cp ).

Case 2: r min ∈ (4r, r]. In that case, we choose r cp = r min ≥ 8r (since r min is a power of 2

V rcp = V + rcp ∪ V - rcp .
Without loss of generality, we can assume that V + rcp ≥ V - rcp . This implies in particular that

r cp C * (ext) k,rcp (V rcp ) ≥ V + rcp V rcp k+rcp-1 k ′ =k m k (V + ) - V + rcp V rcp k-1 k ′ =k-rcp m k (V + ) ≥ 1 2 ⎛ ⎝ k+rcp-1 k ′ =k m k (V + rcp ) - k-1 k ′ =k-rcp m k (V + rcp ) ⎞ ⎠ . Since C * k,rcp (P ) ≥ h 2 and C * (ext) k,rcp (V rcp ) ≤ h 8, this implies that h 4 ≤ C * k,rcp (P ) -2C * (ext) k,rcp (V rcp ) ≤ 1 r cp ⎛ ⎝ k+rcp-1 k ′ =k m k ′ (P ) -m k ′ (V + rcp ) ⎞ ⎠ + 1 r cp ⎛ ⎝ k-1 k ′ =k-rcp m k ′ (V + rcp ) -m k ′ (P ) ⎞ ⎠ ≤ 1 r cp ⎛ ⎝ k-1 k ′ =k-rcp m k ′ (V + rcp ) -m k ′ (P ) ⎞ ⎠ ≤ 1 r cp ⎛ ⎝ k+rcp-1 k ′ =k-rcp m k ′ (V + rcp ) -m k ′ (V - rcp ) ⎞ ⎠ = 2∆ * (ext) k,rcp (V + rcp , V - rcp ) .
Hence, we have proved that ∆ * (ext)

k,rcp (V + rcp , V - rcp ) ≥ h 8 and C * (ext)
k,2rcp (V rcp ) ≥ h 8. Thus, k ∈ D W M (T , P, h, r cp ). We have shown (109) and the proof is finished.

Proof of Lemma F.4. We fix h ∈ H and r ∈ R. Let us consider a subgroup P ⊂ G ∈ L (0,1) (T ) Recall that the blocks Q * W M (P , h, r) = ⋃ r rcp=4r Q * W M (P , h, r, r cp ) -see the definitions in Section F.1. Again, we remove the dependency on T in Q * W M (P , h, r, r cp ) for the ease of exposition. First, we bound ∑ P ∈P * (p) Q * W M (P , h, r, r cp ) before summing over the range over all possible r cp .

Let us consider some l ∈ Q * W M (P , h, r, r cp ). By definition, there exists at least one question k(l) ∈ [lr, (l + 1)r) such that we have simultaneously C * (ext) k(l),2rcp ≥ h 32 and ∆ * (ext)

k(l),rcp ≥ h 32. For l ∈ Q r ∖ Q * W M (r cp )
, we simply define k(l) = l. We deduce from this definition that

Q * W M (P , h, r, r cp ) ≤ l∈Qr 1{C * (ext) k(l),2rcp ≥ h 32}1{∆ * (ext) k(l),rcp ≥ h 32} . (110)
This implies that

Q * W M (P , h, r, r cp ) ≤ 32 h l∈Qr C * (ext) k(l),2rcp ≤ 2 8 r cp rh (111)
where the last inequality comes from the fact that the total variation of m(V 2rcp ) is at most 1 and that, for any l ∈ Q r , the interval [k(l) -2r cp , k(l) + 2r cp ) intersects at most 8r cp r intervals of the form [k(l ′ ) -2r cp , k(l ′ ) + 2r cp ) with l ′ ∈ Q r .

Let p be an integer and assume that P ∈ [p, 2p). Let us introduce Γ ∶= Γ(p, h, r cp ) = r rcp ≥ 1, where we recall that r ≥ 4r 0 is defined by r 0 = 2 9 log(4d R δ) ζ 2 ph 2 in (97). Intuitively, Γ would correspond to the number a + W M and a - W M of sets of experts above P or below P that would be considered if those sets were of size p. More generally, V + rcp (T , P , h) = ∪

a + W M
a=1 G (a) contains at most Γ groups of size at least p among (1) , . . . , G (a W M -1) since the total size of the groups G (a) with a ≤ a W M -1 must be less than 2 11 log(4d R δ) ζ 2 rcph 2 . Thus, we deduce that V - rcp (T , P , h) ∪ P ∪ V + rcp (T , P , h) contains at most 2Γ + 3 groups of size at least p. The following lemma states that the neighbourhoods V - rcp (T , P , h) ∪ P ∪ V + rcp (T , P , h) of groups P in P * (p) only intersect on a few groups.

Lemma F.5. Consider any group P ∈ P * (p). There exists at most 4Γ + 3 groups P

′ ∈ P * (p) such that V - rcp (T , P , h) ∪ P ∪ V + rcp (T , P , h) ∩ V - rcp (T , P ′ , h) ∪ P ′ ∪ V + rcp (T , P ′ , h) ≠ ∅ . (112)
As in the proof of Lemma D.9, we introduce the width of the matrix M on a set A of experts and an interval of questions

[k 1 , k 2 ] by W ∞,1 (M, A, [k 1 , k 2 ]) ∶= max i,j∈A k 2 k=k 1 M i,k -M j,k .
From (110) again, we deduce that

P ∈P * (p) Q * W M (P , h, r, r cp ) ≤ 32 h P ∈P * (p) l∈Qr ∆ * (ext) k(l),rcp (V + rcp (T , P , h), V - rcp (T , P , h)) ≤ 32 h l∈Qr P ∈P * (p) 1 2r cp W ∞,1 (V + rcp (T , P , h) ∪ P ∪ V - rcp (T , P , h), [k -r cp , k + r cp ]) ≤ 32 rh (4Γ + 3)d ,
where the last inequality comes Lemma F.5 and the fact that the sum over disjoints sets

V - rcp (P ) ∪ P ∪ V + rcp (P ) of W ∞,1 (V + rcp (P ) ∪ P ∪ V - rcp (P ), [k -r cp , k + r cp )
) is upper bounded by 2r cp since the total variation of any column of M is at most 1.

Combining (111) with the latter upper bound together with P * (p) ≤ n p we deduce that

P ∈P * (p) Q * W M (P , h, r, r cp ) ≲ nr cp prh ∧ Γd rh . (113)
If r 0 > r, then we have Γ ≤ 8r 0 rcp . This implies that

P ∈P * (p) Q * W M (P , h, r, r cp ) ≲ nr cp prh ∧ r 0 d r cp rh ≲ √ ndr 0 rh √ p .
Since r cp ≤ [4r, r] ∩ R, there are at most c log(d) possible values for r cp , we conclude that

rcp P ∈P * (p) Q * W M (P , h, r, r cp ) ≲ nr 0 prh ∧ r 0 d r 2 h ∧ √ ndr 0 rh √ p .
Otherwise, if r 0 ≤ r, then Γ ≤ 8 and r cp ∈ [4r, 8r]. We deduce from (113) that

P ∈P * (p) Q * W M (P , h, r, r cp ) ≲ n ph ∧ d rh ≤ √ nd √ rh √ p .
We have proved that, in any case,

rcp P ∈P * (p) Q * W M (P , h, r, r cp ) ≲ log(d) ⎡ ⎢ ⎢ ⎢ ⎣ d(r 0 ∨ r) r 2 h ∧ nd(r 0 ∨ r) rh √ p ⎤ ⎥ ⎥ ⎥ ⎦ . ( 114
)
To establish the remaining bound for the sum of Q * W M (P , h, r, r ) , we control each Q * W M (P , h, r, r cp ) individually in a similar fashion to what we did for the analysis of the oblivious hierarchical sorting estimator πHT . First, we have Q

* W M (P , h, r, r cp ) ⊂ Q r so that Q * W M (P , h, r, r cp ) ≤ d r. Besides, arguing as in the proof of Lemma D.2, Q * W M (P , h, r, r cp ) ≲ r cp (rh) ≲ (r 0 ∨ r) (rh). rcp P ∈P * (p) Q * W M (P , h, r, r cp ) ≲ log(d) nd pr ∧ n(r 0 ∨ r) prh . (115)
Combining ( 114) and (115) concludes the proof.

Proof of Lemma F.5. Consider two distinct groups P and P ′ in P * (p). Let (G (a) (P )) a∈Z = Order(T , P ) be the ordering of L (0,1) (T ) centered on P and a ′ ∈ Z the index of the leaf G (a ′ ) (P ) containing P ′ . Obviously, G (a ′ ) ≥ P ′ ≥ p. Without loss of generality, we assume that a ′ > 0. In that case, if (112) is satisfied then necessarily

(V + rcp (T , P , h) ∪ P ) ∩ V - rcp (T , P ′ , h) ≠ ∅ .
This can only happen if the number of leaves G (a) (P ) for 0 < a < a ′ that are of size at least p is less than or equal to 2Γ. The same holds if a ′ < 0 and this proves the lemma.

Appendix G: Proof of Lemma

P n i,[(j-1)l(λ)+1,jl(λ)] ≤ λl(λ) 2 ≤ exp - 3 28 λl(λ) ≤ δ nd ,
provided that λl(λ) ≥ 28 3 log(nd δ). Since λ -≤ 1, we have λl(λ) 2 ≥ Υ * . In view of the definition of Υ * , the condition λl(λ) ≥ 28 3 log(nd δ) is therefore valid and we conclude that

P n i,[(j-1)l(λ)+1,jl(λ)] ≤ Υ * ≤ δ nd ,
and the first result follows. Turning to the second result, we observe that n i,{j} is distributed as a Poisson random variable. We apply again Chernoff's inequality to derive that

P n i,{j} ≤ λ 2 ≤ exp - 3 28 λ ≤ δ nd ,
since λ ≥ 28 3 log(nd δ). Since λ ≥ 2λ -Υ * , the result follows.

G.2. Proof of Theorem 4.3

If λ -≤ 2 d, we use the trivial bound M π-1

W M P -M π * -1 2 F ≤ nd, which ensures that E M π-1 W M P -M π * -1 2 F ≤ n λ - ≤ c log c ′ ndλ 1 2 ζ - n λ .
If λ -≥ 1, then Lemma 4.2 ensures that, with probability higher than 1 -δ, we are able to build the Υ * subsamples and we are in position to apply Theorem 2.3 with subGaussian norm ζ ⌊λ -⌋ 1 2 . Hence, with probability higher than 1 -c ′ log 9 (ndλ

1 2 - (δζ -))δ, we have M π-1 W M P -M π * -1 2 F ≤ c log 11 nd⌊λ -⌋ 1 2 δζ - R F (n, d, ζ⌊λ -⌋ -1 2 ) ≤ c ′ log c ′′ ndλ 1 2 ζ - R F (n, d, ζ⌊λ⌋ -1 2 ) ,
where we use the definition of δ and λ -in the last line. On the complementary event, we simply use that M π-1

W M P -M π * -1 2 F ≤ nd.
Since δ has been chosen small enough, we can conclude that

E[ M π-1 W M P -M π * -1 2 F ] ≤ c ′ log c ′′ ndλ 1 2 ζ - R F (n, d, ζλ - 1 2
) .

It remains to consider the case where λ -∈ [2 d, 1]. Working under the event of probability higher than 1 -δ ensured by Lemma 4.2, we have has been sampled uniformly in {M i,l(λ)(j-1)+1 , M i,l(λ)(j-1)+2 , . . . , M i,l(λ)(j-1)+l(λ) }.

Υ * independent samples Y ↓(0) , . . . , Y ↓(Υ * -1) of size n×⌊d l(λ)⌋. Define the matrix M ↓ of size n×⌊d l(λ)⌋ by M ↓ i,j = M i,l(λ)(j-1)+1 . Obviously, M ↓ π * -1 is a bi-isotonic matrix. Besides, for s = 0, . . . , Υ * -1, (i, j) ∈ [n] × ⌊d l(λ)⌋, we have the decomposition Y ↓(s) ij = M ↓(s) ij + E ↓(s) ij , where M ↓(s) ij belongs to [M ↓ ij , M ↓ ij+1 ] with the convention M ↓(s)
Besides, those are correlated with the noise E ↓(s) ij . For the sake of the analysis, it is in fact easier to consider that M ↓(s) ij has been set by an adversary. Hence, we fall into the semi-random model of Section H and we are in position to apply Theorem H.1 to πW M -SR . With probability at least 1 -c ′ n log 9 ( nd δζ-)δ, we have

M ↓ π-1 WM-SR -M ↓ π * -1 2 F ≤ c log 11 2nd δζ - [R F (n, ⌊d l(λ)⌋, ζ) + n] ,
Define the matrix M ↓↑ of size n × d such that each column is duplicated l(λ) times, except the last one which has been duplicated l(λ) -l(λ)⌊d l(λ)⌋. We readily deduce that

M ↓↑ π-1 WM-SR -M ↓↑ π * -1 2 F ≤ c ′ l(λ) log 11 2nd δζ - [R F (n, ⌊d l(λ)⌋, ζ) + n] , (116)
By triangular inequality, we have

M π-1 WM-SR -M π * -1 2 F ≤ 2 M ↓↑ π-1 WM-SR -M ↓↑ π * -1 2 F + 8 M -M ↓↑ 2 F .
Thus it remains to upper bound the square Euclidean norm of each row of M -M ↓↑ :

d j=1 [M -M ↓↑ ] 2 i,j = ⌊d l(λ)⌋ k=1 l(λ) r=1 [M i,(k-1)l(λ)+r -M i,(k-1)l(λ)+1 ] 2 + d-l(λ)⌊d l(λ)⌋ r=1 M i,(⌊d l(λ)⌋-1)l(λ)+r -M i,(⌊d l(λ)⌋-1)l(λ)+1 2 ≤ ⌊d l(λ)⌋ k=1 l(λ)[M i,kl(λ) -M i,(k-1)l(λ)+1 ] 2 + l(λ)[M i,d -M i,(⌊d l(λ)⌋-1)l(λ)+1 ] 2 ≤ 2l(λ) ,
since the total variation of the i-th row of M is at most one. Hence, M -M ↓↑ 2 F ≤ 2nl(λ). Together with (116), we conclude that

M π-1 WM-SR -M π * -1 2 F ≤ c ′ l(λ) log 11 2nd δζ - [R F (n, ⌊d l(λ)⌋, ζ) + n] .
with probability at least 1 -c ′ n log 9 ( nd δζ-)δ. Since δ has been chosen small enough and since

M π-1 WM-SR -M π * -1 2 F ≤ nd, we conclude that E M π-1 WM-SR -M π * -1 2 F ≤ c ′ l(λ) log 11 2nd ζ - [R F (n, ⌊d l(λ)⌋, ζ) + n] . Since l(λ) ≤ c ′ log c ′′ (nd(λ ∨ 1) ζ -) λ, we deduce from this bound that E M π-1 WM-SR -M π * -1 2 F ≤ c ′ log c ′′ 2nd ζ - ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ζ √ λ 2 ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ nd 1 6 ( ζ √ λ ) 1 3 ∧ n 3 4 d 1 4 ( ζ √ λ ) 1 2 ∧ n √ dλ ∧ n 2 3 √ dλ 1 3 ( ζ √ λ ) 1 3 + n ⎫ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎭ + n λ ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ c ′ log c ′′ 2nd ζ - R F [n, d, ζ √ λ] + n λ , since λ -≤ 1. Again, since λ -≤ 1, we have λ ≤ c 3 log c 4 (nd(λ ∨ 1) ζ)
for some numerical constant c 3 and c 4 . We conclude that

E M π-1 WM-SR -M π * -1 2 F ≤ c ′ log c ′′ 2nd ζ - R F [n, d, ζ √ λ] + n λ e - λ c
D * W M -SR-2 (T , P, h, r, r cp ) = k ∈ 1, . . . , d ∶ m k+rcp (V + rcp ) -m k-rcp (V + rcp ) ≥ hr cp 128 ; (124)
D * W M -SR-3 (T , P, h, r, r cp ) = k ∈ 1, . . . , d ∶ m k+rcp-1 (V - rcp ) -m k-rcp-1 (V - rcp ) ≥ hr cp 128 ; (125)
D * W M -SR-4 (T , P, h, r, r cp ) = k ∈ 1, . . . , d ∶ m k+2rcp (V 2rcp ) -m k-2rcp-1 (V 2rcp ) ≥ hr cp 128 . (126)
Then, we define the corresponding subsets

Q * W M -SR-1 , Q * W M -SR-2 , Q * W M -SR-3 , and Q * W M -SR-4 of Q r . For short, we write Q * W M -SR = Q * W M -SR-1 ∪ Q * W M -SR-2 ∪ Q * W M -SR-3 ∪ Q * W M -SR-4 .
We have the following counterpart of Lemma F.1.

Lemma H.4. Consider any valid hierarchical sorting tree T , any subset P of a leaf G of T , any h ∈ H, and any r ∈ R. With probability at least 1 -δ, it holds that

Q * W M ⊂ QW M ⊂ Q * W M -SR . (128)
Obviously, Lemma F.2 is still true since it does not depend on the data generating process. Then, we adapt Propositions D.4 and D.5 to this adversarial setting. Proposition H.5. Consider any P ⊂ [n], any r ∈ R, and any subset Q ⊂ Q r . Also, fix any η > 0 and any φ > 0. Provided that

Θ(P , Q) -Θ(P , Q) η 2 F ≥ 1 φ Θ(P , Q) -Θ(P , Q) 2 F ≥ 8η P φ l 1 log( 2 P δ ) Q + 20 ,
then, with probability higher than 1 -δ, we have

Θ(P ′ , Q) -Θ(P ′ , Q) 2 F ≤ 1 - 1 16φ Θ(P , Q) -Θ(P , Q) 2 F .
Recall the definition (60) of φ l 1 . Henceforth, the matrix

Θ( P , Q) is said to be indistinguish- able in l 1 -norm if it satisfies max i,j∈P Θ i,⋅ ( P , Q) -Θ j,⋅ ( P , Q) 1 ≤ φ l 1 Q log 2 P δ + 20 . ( 129
) Proposition H.6. Let P ⊂ [n] and Q ⊂ [d]. If Θ( P , Q) is indistinguishable in l 1 -norm and if Θ( P , Q) -Θ( P , Q) 2 F ≥ 10 6 log 3 6nd δζ - ζ 2 P Q + P + P , (130)
then, with probability higher than 1 -3δ, we have

Θ(P ′ , Q) -Θ(P ′ , Q) 2 F ≤ 1 - 1 200 log 2 (nd ζ -) Θ(P , Q) -Θ(P , Q) 2 F .
Equipped with these two propositions, we arrive at the counterpart of Propositions D.7 and F.3. Recall Definition (64) of the function Ψ by Ψ(p, r, h, q) = hp √ rq ζ ∧ √ pq + p. Proposition H.7. Assume that T t is a valid hierarchical sorting tree. Consider a leaf G of T of type 0 or 1 at depth t. With probability higher than 1 -5τ ∞ δ, there exists a subset P † such that P ⊆ P † ⊆ G and the following property holds. For some r † cp ≥ r † ∈ R and some h † ∈ H, upon writing

Q † W M = Q * W M and Q † W M -SR = Q * W M -SR , we have simultaneously Θ(P † , Q † W M -SR ) -Θ(P † , Q † W M ) √ r † h † 2 F ≤ 2 ⋅ 10 6 log 3 6nd δζ - ζ 2 Ψ( P † , r † , h † , Q † W M -SR ) + P † ; (131)
M (P † ) -M (P † ) 2 F ≤ 16ζ 2 + 96 R H [Θ(P † , Q † W M ) -Θ(P † , Q † W M )] √ r † h † 2 F . ( 132
)
The proof is analogous to that of Proposition F.3, up to some numerical constants, and is omitted.

Then, we state the counterpart of Lemma F.4 to control Q † W M -SR . In comparison to this lemma, we have an additional term n (prh).

Lemma H.8. Assume that T t is a valid hierarchical sorting tree. For any h ∈ H, r ∈ R, any integer p, any sequence P v of subsets of G v , it holds that

P ∈P * (p) Q * W M -SR (P , h, r) ≲ log(d) ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ nd(r 0 ∨ r) rh √ p ∧ d(r 0 ∨ r) r 2 h ∧ nd pr ∧ n(r 0 ∨ r) prh ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ + n prh ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ . (133)
Then, we apply Proposition H.7 to control the loss on an additional event of probability higher than 1 -5 ⋅ 2 t τ ∞ δ.

P ∈Lt M (P ) -M (P ) 2 F ≤ 16ζ 2 L t + 96 R H p,h,r P † ∈P * (p,h,r) Θ(P † , Q † W M ) -Θ(P † , Q † W M ) √ rh 2 F ≲ log 5 6nd δζ -p,h,r P † ∈P * (p,h,r) ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ζ 2 [ h 2 pr ζ 2 ∧ 1]p Q * W M (P † , h, r) + (ζ 2 ∨ 1)p ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≲ log 5.5 6nd δζ -p,h,r ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ζ 2 nr r ∨ r 0 P † ∈P * (p,h,r) Q * W M (P † , h, r) + (ζ 2 ∨ 1)n ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ≲ log 6 6nd δζ -p,h,r ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ζ 2 ⎛ ⎝ n 3 4 d 1 4 1 (r 0 ∨ r)ph 2 1 4 ∧ n d p(r 0 ∨ r) ∧ n √ ph ∧ nd rh ⎞ ⎠ + ζ 2 n 1 pr 0 h + (ζ 2 ∨ 1)n ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ (a) ≲ log 7 6nd δζ -p,h ζ 2 n 3 4 d 1 4 ζ 1 2 ∧ n √ d ∧ nh ζ √ d ∧ n √ h ∧ √ nd √ h + (ζ 2 ∨ 1)n ≲ log 9 6nd δζ - ζ 2 n 3 4 d 1 4 ζ 1 2 ∧ n √ d ∧ n 2 3 √ d ζ 1 3 ∧ nd 1 6 ζ 1 3 + (ζ 2 ∨ 1)n ,
where, in (a), we use that pr 0 h ≥ pr 0 h 2 ≳ 1, the rest of the bounds being analogous to the proof of Proposition H.3. This concludes the proof.

H.4. Proofs of the lemmas

Proof of Lemma H.4. By a union bound and arguing as in the proof of Lemma F.1, we deduce that, with probability higher than 1 -δ, we have simultaneously

max rcp∈[4r,r]∩R max k∈[d] Ĉ * (ext) k,2rcp -E Ĉ * (ext) k,2rcp ≤ h 32 ; (134) max rcp∈[4r,r]∩R max k∈[d] ∆ * (ext) k,rcp -E ∆(ext) k,rcp ≤ h 32 . (135)
Because of the adversarial observations, we now have

C * (ext) k,2rcp ≤ E Ĉ * (ext) k,2rcp ≤ C * (ext) k,2rcp + m k+2rcp (V) -m k-2rcp-1 (V) 2r cp , ∆ * (ext) k,rcp ≤ E ∆ * (ext) k,rcp ≤ ∆ * (ext) k,rcp + m k+rcp (V + ) -m k-rcp-1 (V -) 2r cp + m k+rcp-1 (V + ) -m k-rcp (V -) 2r cp .
Combining the above bounds with (134) and (135) allows us to conclude.

Proof of Proposition H.5. With the notation of the proof of Proposition D.4, the condition ( 71) is now replaced by max i,j∈P

′ Θ(P ′ ) i,⋅ -Θ(P ′ ) j,⋅ 1 ≤ Φ l 1 Q + 20 , (136)
where we used Lemma H.2 with w = 1 Q . The rest of the proof is left unchanged except that we replace

Φ l 1 Q by Φ l 1 Q + 20.
Proof of Proposition H.6. Lemma E.4 is still true. However, Lemma E.5 needs to be updated to Lemma H.9. Fix any δ ∈ (0, 1).

If Θ -Θ 2 op ≥ 6400 P + ζ 2 Q P + log(6 δ)) + 7 P + 2 log(6 δ) , (137)
then, with probability higher than 1 -δ, we have

vT Θ -Θ 2 2 ≥ 1 2 Θ -Θ 2 op .
In light of Condition (130), this assumption is valid. Together with Lemma E.4, we deduce that there exists an event of probability higher than 1 -δ such that

vT Θ -Θ 2 2 ≥ 1 2 Θ -Θ 2 op ≥ 1 32 log 2 (nd ζ -) Θ -Θ 2 F .
As the vectors ẑ and ŵ are defined though Z (3) , we rather focus on Θ (3) . By (121), we have

Θ (3) -Θ op ≤ P . vT (Θ (3) -Θ (3)
)

2 2 ≥ vT (Θ -Θ) 2 2 -4 Θ -Θ (3) op Θ -Θ op ≥ vT (Θ -Θ) 2 2 -4 P Θ -Θ op ≥ 9 20 Θ -Θ 2 op ≥ 1 36 log 2 (nd ζ -) Θ -Θ 2 F . (138)
Then, the analysis of ẑ and z * follows the same steps as in the original proofs, -see Appendix E.3 -the main difference being that we invoke (136) instead of (71). More precisely, we still have

vT (Θ (3) -Θ (3)
)

ŵ ŵ 2 2 ≥ 16 25 w * 2 2 . (139)
In light of Condition (62), this quantity is small compared to Θ -Θ 2 F .

(Θ(P

′ ) -Θ(P ′ )) ŵ+ ŵ+ 2 2 2 ≤ 1 200 log 2 (nd ζ -) Θ -Θ 2 F . (142)
Then, we conclude from (142) as we did from (83) in the original proof.

Proof of Lemma H.9. For short, we write p = P . Since Z (t) = Θ (t) + N (t) for t = 1, 2, the difference wih Lemma E.5 is that Θ (1) and Θ (2) are involved in the terms v T (Z (1) -Z

) and v T (Z (2) -Z

). Hence, arguing as in the proof of Lemma E.5, we derive that, on an event of probability higher than 1 -3δ, we have simultaneously for all v ∈ R p with v 2 ≤ 1 that v T (Z (1) -Z

)

2 2 -v T (Θ (1) -Θ (1)
)

2 2 + 1 2 v T (Θ (1) -Θ (1)
-

Θ (2) + Θ (2)
) 2 2 -

1 2 v T (Z (1) -Z (1)
-

Z (2) + Z (2)
)

2 2 ≤ 10ζ Θ (1) -Θ (1) op + 1 2 Θ (1) -Θ (1)
-Θ (2) + Θ

(2) op 2p + log(6 δ)

+ 192ζ 2 q(3p + log(6 δ)) + (3p + log(6 δ)) .
By (121), we have Θ (t) -Θ op ≤ √ p for t = 1, 2. Hence, the above bound simplifies in

v T (Z (1) -Z (1)
)

2 2 -v T (Θ -Θ) 2 2 - 1 2 v T (Z (1) -Z (1)
-

Z (2) + Z (2)
)

2 2 ≤ 10ζ Θ -Θ op + 4 √ p 2p + log(6 δ) + 192ζ 2 q(3p + log(6 δ)) + (3p + log(6 δ)) + 12p + 4 √ p Θ -Θ op .
Since we assume that Θ -Θ 2 op ≥ 6400 p + ζ 2 [ q(5p + log(6 δ)) + 7p + 2 log(6 δ)] , we deduce that, on the same event, we have

sup v∈R p ∶ v 2 ≤1 v T (Z (1) -Z (1)
)

2 2 -v T (Θ-Θ) 2 2 - 1 2 v T (Z (1) -Z (1)
-Z (2) +Z

) 2 2 ≤ 1 4 Θ-Θ 2 op . (2)
The rest of the proof is left unchanged.

Proof of Lemma H.8. Recall that Q * W M -SR (P , h, r) decomposes as the union of

Q * W M -SR-1 , Q * W M -SR-2 , and Q * W M -SR-3 , Q * W M -SR-4 . Since Q * W M -SR-1 is defined analogously to Q * W M
-but with a different numerical constant-, we can argue as in the proof of Lemma H.8, which yields

P ∈P * (p) Q * W M -SR-1 (P , h, r) ≲ log(d) ⎡ ⎢ ⎢ ⎢ ⎣ nd(r 0 ∨ r) rh √ p ∧ d(r 0 ∨ r) r 2 h ∧ nd pr ∧ n(r 0 ∨ r) prh ⎤ ⎥ ⎥ ⎥ ⎦ .
It remains to consider the three last sets. We only focus on Q * W M -SR-2 (P , h, r), the last ones being analogous. We first focus on a single set Q * W M -SR-2 (P , h, r, r cp ). If k belongs to D * W M -SR-2 (P , h, r, r cp ), this implies that the total variation of m(V + rcp ) between kr cp and k + r cp is at least hr cp 128. Since the total variation of m(V + rcp ) is at most one, there are at most c (hr cp ) regions of Q rcp that contain at least a point D * W M -SR-2 (P , h, r, r cp ), which entails that there are at most c ′ rcp r ⋅

Q * W M -SR-2 (P , h, r, r cp ) ≲ log(d) n prh ,
which concludes the proof.

Appendix I: Proofs for the l ∞ loss Proof of Lemma 4.5. Without loss of generality, we assume that π * is the identity. Fix any i ∈ [n] and assume that π-1 (i) ≠ i. Consider for instance the case where m = π-1 (i) > i. As a consequence, there are at least l = m-i experts that are below m in the oracle order and above m in the estimated order . Denote j the smallest of those experts. Hence, we have j ≤ i < m and π(j) ≥ π(m). Besides, since j ≤ i ≤ m, we deduce from the bi-isotonic assumption that

M i,. -M π-1 (i),. 2
2 ≤ M j,. -M π-1 (i),.

2 2 = M j,. -M m,. 2 2 .
Taking the supremum over all i implies that l ∞ (π, π * ) ≤ l err (π, π * ). Let us turn to the second inequality. Consider any i < j such that π(i) > π(j). We consider three subcases.

(i) If π(i) ≥ j, then we have M i,. -M j,. (iii) It remains to consider the case where we have i < π(j) < π(i) < j. As a consequence, for each k ∈ [d], we have M j,k -M i,k ≤ M j,k -M π(j),k + M π(i),k -M i,k , which in turn implies that M j,. -M i,. Taking the supremum over all i and reminding the definition of j concludes the proof.

Proof of Proposition 4.6. For n = 2, all the losses are equal. Hence, the minimax lower bound [START_REF] Zhang | Heteroskedastic pca: Algorithm, optimality, and applications[END_REF] is a straightforward consequence of the general minimax lower bound of Theorem 4.1 by a reduction to the case where n = 2 (recall that ζ = 1 here) -This reduction is achieved by putting to 0 the signal corresponding to all n -2 experts that do not corresponds to the 2 experts of interest that will be most difficult to distinguish so that estimating the permutation amounts to deciphering between these two experts. Hence, we derive that

inf π sup π * ∈Πn sup M ∶ M π * -1 ∈C BISO E (π * ,M ) [l ∞ (π, π * )] ≥ c ′′ d 1 6 λ 5 6 ⋀ √ d λ + 1 λ ⋀ d .
It turns out that the term 1 λ is higher than d if λ ≤ 1 d and is smaller than d Regarding the upper bound, we build upon the analysis of πW M P in the specific case of n = 2. Consider any fixed i and j. With probability higher than 1-cδ log c ′ [nd(λ∨1)], it follows from the proof of Theorems 2.3 and 4.3 that (i) πW M P builds a valid hierarchical sorting tree and (ii) the set P ⊂ {i, j} built at the end of BlockSort satisfies M (P ) -M (P ) 2 F ≤ c 1 log c 2 (nd(λ ∨ 1))

d 1 6 λ 5 6 ⋀ √ d λ + 1 λ . ( 143
)
It follows from (i) that (i, j) (resp. (j, i)) is added to PC only if π * (i) < π * (j) (resp. π * (i) < π * (j)). Besides, if M i,. -M j,. 2 > 2c 1 log c 2 (nd(λ ∨ 1))

d 1 6 λ 5 6 ⋀ √ d λ + 1 λ , (144)
then, this implies that P ≤ 1, otherwise this would contradict Equation (143). Then, taking a union bound over all possible (i, j), we deduce that there exists an event of probability higher than 1 -cn 2 δ log c ′ (nd(λ ∨ 1)), such that PC is consistent and contains all 2-tuples of experts that satisfy (144).

Turning to the estimated permutation πP C , we consider any two experts such that π * (i) < π * (j) and πP C (i) > πP C (j). The latter condition implies that φ(i) ≥ φ(j). Since PC is consistent, we have π * (i) ≥ 1 + φ(i). Define π * -(j) as the number of experts k that are below j and are far apart from j in the sense of Equation (144). We know that, under the above event, we have that φ(j) ≥ π * -(j). This implies that π * (i) > π * -(j). As a consequence, i and j are not far apart in the sense of Equation ( 144). This implies that l err (π P C , π *

) ≤ 2c 1 log c 2 (nd(λ ∨ 1))

d 1 6 λ 5 6 ⋀ √ d λ + 1 λ .
Since l err (π P C , π * ) is equivalent to l ∞ (π P C , π * ), this bound also holds (with a larger constant) for the latter loss. Since δ has been chosen small enough and since the loss is always smaller than d, we arrive at the following risk bound We build a prior distribution ν of M as follows. For each row i = 1, . . . , n, we sample W i ∼ B(1 2). If ζ i = 1, the i-th row of M is constant and equal to 1. if W i = 0, then the i-th row of M has its d -first entries equal to 0, while the remaining entries are equal to 1.

We write P and E for the corresponding marginal probability and expectations of the data

(x t , y t ). R * [n, d, λ, 0] ≥ inf π E M π-1 -M π * -1 2 2 .
For each entry i = 1, . . . , n, we write N i = ∑ t 1 xt∈{i}×[d-] the number of observations on the d - first columns of the i-th row. If N i ≥ 1, then the statistician knows the value of W i . Conversely, if N i = 0, then she has no information on the value of W i . Given an estimator π, it is always possible to reduce its loss by ranking at the top the experts such that N i ≥ 1 and W i = 1, ranking below the experts such that N i ≥ 1 and W i = 0, and putting in between the experts such that N i = 0. Conditionally to the observations (x t , y t ), the values of W i such that N i = 0 are still distributed according to a Bernoulli distribution. As a consequence, for any π which has been rearranged as explained above, the conditional risk satisfies

E M π-1 -M π * -1 2 2 (x t , y t ) ≥ d -× g( n i=1
1

N i =0 ) ,
where g(k) corresponds to the expected number of error of π when there are exactly k rows without any observations. Since conditionally to π, the corresponding values of W have been sampled independently as Bernoulli random variables with parameter 1 2, we arrive at the following expression for g(k):

g(k) = k i=1 P[{W i = 1} ∩ { k j=1 W j ≤ k -i}] + P[{W i = 0} ∩ { k j=1 W j > k -i}] .
We have g(1) = 0, g(2) = 1 2, g(3) = 1. For k ≥ 4, we focus on the ⌊k 4⌋ first and ⌊k 4⌋ last entries to deduce that

g(k) ≥ E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ⌊k 4⌋ i=1 W i ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ P ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ k i=⌊k 4⌋+1 W i ≤ k 2 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ + E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ k i=k-⌊k 4⌋+1 (1 -W i ) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ P ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ k-⌊k 4⌋ i=1 (1 -W i ) ≤ k 2 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≥ 0.5⌊ k 4 ⌋ .
Hence, there exists a universal constant c > 0 such that we have g(k) ≥ c(k -1) for any k ≥ 1. Since N i follows a Poisson distribution with parameter λd -, V = ∑ n i=1 1 N i =0 follows a binomial distribution with parameters (e -λd-, n). We obtain R * e -2λd-(1 -e -2λd-) n-2 ≥ c ′ n 2 e -2λd-. In any case, we conclude that R * [n, d, λ, 0] ≥ c ′′ d -ne -2λd-. If λ ≤ 1 d, then d -= d, and the right hand-side is higher than c ′′ nde -2 . If λ ∈ [1 d, 1], then we have d -∈ [1 (2λ), 1 λ] and the right hand-side risk is higher than cn λ. Finally, if λ ≥ 1, we take d -= 1 and the right hand-side is higher than c ′ ne -2λ . We have proved Equation (145).

J.1.2. Proof of the remaining regimes

Since the minimax risk is increasing with n and d, we can assume without loss of generality that both n and d express as a power of 2.

We shall first build a collection of prior distributions ν G indexed by G ∈ G on M . We denote P (full) G and E (full) G the corresponding marginal probability distributions and expectations on the data (x t , y t ). Since we aim at proving the lower bound in the Gaussian setting, we assume that the data y t is a normal random variable with mean M xt and variance ζ 2 conditionally on M and x t . The minimax risk (4) is higher than the worst Bayesian risk.

R * [n, d, λ, ζ] ≥ inf π sup G∈G E full G M π-1 -M π * -1 2 F . (146)
We first spend some time defining the corresponding prior distributions before applying a sequence of reduction arguments.

other, it turns out that the loss of π is smaller than that of π. Ĝ(1) ≠ G t (1) .

For any group G t (1) , under ν G t(1) , the rows of the restrictions of M to G To simplify the notation, we write henceforth G and Ĝ for G t (1) and Ĝ(1) respectively. We also write P G for the corresponding marginal distribution of N and Y ↓ . By a sufficiency argument, it turns out that inf Ĝ sup

G P (full) G Ĝ ≠ G = inf Ĝ sup G P G Ĝ ≠ G .
Hence, we arrive at the following conclusion

R * [n, d, λ, ζ] ≥ nqυ 2 dζ 2 8λ 0 d inf Ĝ sup G P G Ĝ ≠ G . (151)
Let us introduce a third-part distribution P 0 on N and Y ↓ corresponding to the case υ = 0. Each of the entry of N therefore follows an independent Poisson distribution with parameter λ and, given N i,j , we have Y ↓ ∼ N (0, N i,j ζ 2 ). We then deduce from Fano's inequality [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] that

inf Ĝ sup G∈G P G ( Ĝ ≠ G) ≥ 1 - 1 + max G∈G KL(P G P 0 ) log( G ) , (152)
where KL(. .) stands for the Kullback-Leibler divergence. Then, the following lemma bounds these Kullback-Leibler divergences.

Lemma J.2. Assume that λ 0 = ñλd d ≥ 1 and that 8υ 2 ≤ 1. For any G ∈ G, we have KL(P G P 0 ) ≤ 4υ 2 q 2 d .

In the specific case where d = q = 1, we have KL(P G P 0 ) = υ 2 2 for any G ∈ G, any λ 0 > 0, and any υ > 0.

Let us summarize our findings by combining (151), (152), with Lemma J.2 and the different constraints on the parameters (149).

Proposition J.3. Provided that ñ, d, q, and υ satisfy the two following conditions λ ≥ d ñd ;

(153)

υ ≤ 2 -3 2 ⋀ ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ c 0 dñ q ⋀ c 1 √ λ ζ ñ3 2 d 1 2 n d1 2 ∧ √ ñd d3 2 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ , (154)
then, we have

R * [n, d, λ, ζ] ≥ c ′′ nqυ 2 ζ 2 ñλ . (155)
In the specific case where we fix d = q = 1 and ñ = n, we can deduce from combining (151), (152), and the second part of Lemma J.

Since we have proved the lower bound (145) and (156), we only have to prove the corresponding minimax lower bound for the remaining four rates. For this purpose, we shall fix the values of ñ, d, q, and υ and apply from Proposition J.3. In the sequel we write ⌊x⌋ dya for 2 ⌊log 2 (x)⌋ .

Case 1: Rate nd 1 6 λ ′5 6 . This rate can only occur if n ≤ d, λ ′ ∈ [n 3 d, d 2 ] and λ ≥ 1 ∧ [λ ′5 6 d 1 6 ]. In this case, we take ñ = 2, d = ⌊(λ ′ d) 1 3 ⌋ dya , and q = ⌊ d⌋. One readily checks that the conditions (153) and (154) are satisfied for a universal numerical value of υ. Then, Proposition J.3 leads to the desired rate. In this case, we take d = ⌊(λ ′ nd) 1 4 ⌋ dya , ñ = ⌊n d⌋ dya , and q = ⌊ ñ d⌋. One readily checks that the conditions (153) and ( 154) are satisfied for an universal numerical value of υ. Then, Proposition J.3 leads to the desired rate. In this case, we take ñ = ⌊(n 2 λ ′ ) 1 3 ⌋ dya , d = d, and q = ⌊ ñ d⌋. One readily checks that the conditions (153) and (154) are satisfied for an universal numerical value of υ. Then, Proposition J.3 leads to the desired rate. ∑ i∈G Y ↓ i,j are a sufficient statistic for deciphering P G and P 0 , we have KL(P G P 0 ) = KL(P ′ P) where P ′ and P stand for the corresponding marginal distributions of N and Y ↓ .

Set u = υ √ λ 0 . Under P, given N , the Y ↓ j 's are independent and satisfy Y ↓ j ∼ N (0, N j ). Under P ′ , conditionally to the subset Q of size q and conditionally to N , the Y ↓ j 's are independent and satisfy Y ↓ j ∼ N (uN j 1{j ∈ Q}, N j ).

In the specific case of q = d = 1, we can explicitely compute the Kullback Leibler divergence. Conditionally to N 1 = x, Y

↓ is either distributed N (0, x) under P and N (ux, x) under P ′ . Hence, their conditional Kullback-divergence is u 2 x 2. Integrating with respect to x, we conclude that

KL(P ′ P) = E u 2 2 N = u 2 λ 0 2 = υ 2 2 .
We have shown the second result.

Let us come back to the general case. For z = 1, 0, define

α z (x, y) = λ x 0 e -λ 0 x! 1 √ 2αx exp(- (y -uxz) 2 2x
) .

Then, the density of P with respect to µ ⊗ λ where µ is the discrete measure and λ is the Lebesgues measure is ∏ j α 0 (N j , Y ↓ j ). Besides, the density of P ′ is

j α 1 j∈Q (N j , Y ↓ j ) dη(Q) ,
where η stands for the uniform distribution over {Q ∈ {0, 1} d ∶ Q 0 = q}. It is more convenient to first control the χ 2 distance P and P ′ . Since this distance is, up to an additive term of order 1, the second moment of the likelihood ratio between P and P ′ , we arrive at the following χ 2 (P ′ , P) + 1

= j∈Q∩Q ′
[α 1 (x j , y j )] 2 α 0 (x j , y j ) dµ(x j )dy j j∈Q∆Q ′ α 1 (x j , y j )dµ(x j )dy j dη(Q)dη(Q ′ ) = j∈Q∩Q ′

[α 1 (x j , y j )] 2 α 0 (x j , y j ) dµ(x j )dy j dη(Q)dη(Q ′ ) , since α 1 is a density. Let us work out each of these ratios. Coming back to the χ 2 distance, we arrive at the following equality

χ 2 (P ′ , P) = exp I Q ∩ Q ′ dη(Q)dη(Q ′ ) -1 .
Here, Q ∩ Q ′ is distributed as an Hypergeometric distribution with parameters d, q, and q d. We know from Aldous (p.173) [START_REF] Aldous | Exchangeability and related topics, École d'été de probabilités de Saint Flour[END_REF] that Q ∩ Q ′ follows the same distribution as the random variable E(W B) where W is a binomial random variable of parameters q, q d and B is some suitable σ-algebra. By Jensen's inequality, we deduce that χ 2 (P ′ , P) ≤ E[exp(IW )] -1 = 1 + q d (exp(I) -1) q -1 .

Recall that λ 0 u 2 = υ 2 ≤ 1 8. Hence, provided that λ 0 = ñλd d ≥ 1, we have I ≤ 2λ 0 u 2 = 2υ 2 . It then follows that χ 2 (P ′ , P) ≤ exp q 2 d(exp(I) -1) -1 ≤ exp 4υ 2 q 2 d -1.

To conclude, we use the classical bound KL(P ′ P) ≤ log 1 + χ 2 (P ′ , P) -see e.g. [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. This leads us to KL(P ′ P) ≤ 4υ 2 q 2 d .

J.2. Proof of Theorem 2.1

Fix n, d, ζ, and κ ≥ 2, and assume that, for some c ′ , there exists an estimator π satisfying

sup π * ∈Πn sup M ∶ M π * -1 ∈C BISO E (π * ,M ) M π-1 -M π * -1 2 F ≤ c ′ [log -κ (nd ζ -)R F [n, d, ζ] ⋀ nd] , (158)
with Υ = ⌊log κ (nd ζ -)⌋ samples. Let us show that this bound would contradict the minimax lower bound in the Poisson setting. Fix λ = 112 3 log κ (nd ζ -) and consider the model [START_REF] Ballinger | Decisions, error and heterogeneity[END_REF]. Define the estimator π such that π = π under the event A such that there are at least Υ observations on each entry and π is computed arbitrarily otherwise. By (158), π satisfies

E (π * ,M ) M π-1 -M π * -1 2 F ≤ ndP[A c ] + c ′ [log -κ (nd ζ -)R F [n, d, ζ] ⋀ nd] . (159)
By Chernoff'inequality for Poisson random variable, we deduce that

P[A c
] ≤ nd exp - There exists a constant c 0 such that for any κ ≥ 2, e -4x κ +2x ≤ c 0 x 2κ . We deduce that

P[A c ] ≤ ζ 2 - nd c 0 log 2κ (nd ζ -)
,

where we used that e x ≥ 1 + x β β for any x ≥ 0 and any β > 0 and that κ ≥ 2. We then deduce from (159) that

E (π * ,M ) M π-1 -M π * -1 2 F ≤ c ′ + c 0 log κ (nd ζ -) [log -κ (nd ζ -)R F [n, d, ζ] ⋀ nd] . ( 160
) For λ ≥ 1, R F [n, d, ζ √ λ] ≥ R F [n,d,ζ] λ
. We deduce that

E (π * ,M ) M π-1 -M π * -1 2 F ≤ 112 3 c ′ + c 0 log κ (nd ζ -) R F [n, d, ζ √ λ] ⋀ nd .
Taking c ′ small enough compared to the numerical constant c in Theorem 4.1 contradicts this last theorem provided that nd ζ -is larger than some some numerical constant. Hence, no estimator can achieve (158) for this constant c ′ when (nd ζ -) is large enough.

It remains to consider the case where nd ζ -is smaller than some constant c ′′ ≥ 2. We only need to prove that the minimax risk is lower bounded by

n ≤ d 1 3 d 1 3 1 ] nd 1 6 n 3 4 d 1 4 nFigure 1 :

 3141 Figure 1: Summary of the minimax risks (up to poly-logarithmic terms) for permutation estimation (R * [n, d, 1, 1]) and matrix estimation (R * est [n, d, 1, 1]) in the specific cases where λ, ζ = 1.

Figure 2 :

 2 Figure 2: Example of a hierarchical sorting tree.

Figure 3 :

 3 Figure 3: In this toy example, the group P is only made of two types of experts, those in U * and those in L * . The high-variation region of m(P ) corresponds to the questions on which U * and L * differ.

Figure 4 :

 4 Figure 4: In this toy example, the group P is only made of two types of experts U * and L * with L * = U * = 3.

Figure 5 :

 5 Figure 5: In these two panels, the group P is only made of two types of experts, those in U * and those in L * . The curves m(V + rcp) and m(V - rcp ) respectively correspond to the expected average experts in the neighboring groups V + rcp and V - rcp defined in[START_REF] Pananjady | Isotonic regression with unknown permutations: Statistics, computation and adaptation[END_REF]. In the left panel, the third and fourth blocks are not selected because the corresponding statistic ∆(ext) k,rcp is small. In the right panel, both the statistic Ĉ(ext) k,rcp and ∆(ext) k,rcp are large compared to h. If P is small, this block is selected using a scale r cp ≲ r.

Theorem 4 . 1 .

 41 There exist universal constants c and c ′ such that the following holds for any n ≥ 2, any d ≥ 1, λ > 0, and ζ ≥ 0:

  ), we recognize two terms. First, R F [n, d, ζ √ λ] corresponds to the minimax risk for recovering π * in a full observation model with noise ζ √ λ. The second term n λ e -2λ

  Acknowledgements. The work of A. Carpentier is partially supported by the Deutsche Forschungsgemeinschaft (DFG) Emmy Noether grant MuSyAD (CA 1488/1-1), by the DFG -314838170, GRK 2297 MathCoRe, by the FG DFG, by the DFG CRC 1294 'Data Assimilation', Project A03, by the Forschungsgruppe FOR 5381 "Mathematical Statistics in the Information Age -Statistical Efficiency and Computational Tractability", Project TP 02, by the Agence Nationale de la Recherche (ANR) and the DFG on the French-German PRCI ANR ASCAI CA 1488/4-1 "Aktive und Batch-Segmentierung, Clustering und Seriation: Grundlagen der KI" and by the UFA-DFH through the French-German Doktorandenkolleg CDFA 01-18 and by the SFI Sachsen-Anhalt for the project RE-BCI. The work of E. Pilliat and N. Verzelen has been partially supported by ANR-21-CE23-0035 (ASCAI).

4 :

 4 (Lpca, Upca), (Lpca, U pca) = DoubleTrisection -PCA(Z ′ , γ) 5: Set L = Lcp ∪ Lpca and L = Lcp ∪ Lpca and U = Ucp ∪ Upca and U = U cp ∪ U pca 6: return (L, U ), (L, U )

Algorithm 10

 10 BlockSort(Y, T , G) Require: 6τ∞ samples Y = (Y (0) , . . . , Y (6τ∞-1) ), the tree T , a leaf G ⊂ [n] in T at maximal depth Ensure: A partition of G into three groups (O, P, I) 1: Set γ = ⌊ G 2⌋ and O0, I0, O0, I0 = ∅ 2: for τ = 0, . . . , τ∞ -1 do 3: Consider 6 fresh samples Yτ = (Y (6τ ) , . . . , Y (6τ +5) ) 4: set γ = ⌊ G 2⌋ -Oτ 5:

Definition 2 .

 2 (Valid hierarchical sorting Tree) A hierarchical sorting tree T is valid if nonterminal nodes G of T satisfy the two following additional properties: if we denote (O, P, I) their children of type 0, p, 1 respectively, then • All the experts in O are below those of I. In other words, for any i ∈ O and any j ∈ I, we have π * (i) < π * (j). • O < G and I < G .

7 :

 7 , a set of experts P , h ∈ H and r in R Ensure: An encoded set including the high-variation regions Qcp ∶= Qcp(Y, P , h, r) ⊂ Qr 1: y(P ) = 1 P ∑ i∈P Yi,⋅ 2: r = 8 ⌈ 32ζ 2 P h 2 log( 2d δ ) ⌉ ∨ r 3: Initialize Dcp = ∅ 4: for k ∈ [d] do 5Dcp = {k ∈ [d] ∶ Ĉk (y(P )) ≥ h 4} 8: Qcp = Encode -Set( Dcp, r) 9: return Qcp

( 1 . 1 .

 11 O, P, I) = (O τ∞ , G ∖ (O τ∞ ∪ I τ∞ ), I τ∞ ) is returned by the algorithm. The second one (O, P , I) = (O τ∞ , G ∖ (O τ∞ ∪ I τ∞ ), I τ∞ ) is important for our analysis. From the definitions of the different procedures, one readily checks that O ⊂ O and I ⊂ I. In fact, we shall prove later that, with high probability, the subsets (O, P, I) and (O, P , I) satisfy the following stronger property. Property {O, P, I} and {O, P , I} are partitions of the leaf G with O ⊂ O, I ⊂ I, and P ⊂ P , 2. For ω = π * -1 {O,P,I} π * {G} , we have ω(i) = i for any i ∈ O ∪ I. 3. For any i ∈ O and j ∈ I, we have π * (i) < π * (j). 4. We have O ≤ G 2 and I ≤ G 2.

  The sets (O, O, I, I) built in BlockSort arise as unions of set (L, U ) and (L, U ) that are computed by DoubleTrisection for a set P and a pivot γ ∈[1, P ]. For this reason, we now state a desired property of the result of the algorithm that will enforce Property 1.Property 2 (Property on (L, U ) and (L, U )). For P′ = P ∖ (L ∪ U ) and P ′ = P ∖ (L ∪ U ), we have 1. L ⊂ L, and U ⊂ U , 2. if ω = π * -1{L,P ′ ,U } π * {P } then for any i ∈ L ∪ U it holds that ω(i) = i, 3. For any i ∈ L and j ∈ U we have π * {P } (i) < γ < π * {P } (j). We say that DoubleTrisection with (Y, T , P , γ) satisfies Property 2 if the corresponding subsets (L, U ) and (L, U ) satisfy Property 2. Proposition B.2 (Deterministic Analysis of BlockSort). BlockSort satisfies Property 1 on (Y, T , G) if, at each step of Algorithm 10, each call of DoubleTrisection satisfies Property 2.

  Propositions B.1-B.3 are relatively simple. The main difficulty and the key arguments lie in the proofs of Proposition B.5 and B.6 which are respectively in Sections D and F. Appendix C: Proofs of Propositions B.1, B.2, and B.3

2 .

 2 Consider any data Y, any tree T and any leaf G of T . Let (O, P, I) and (O, P , I) denote the trisections built in BlockSort(Y, T , G). For any τ < τ ∞ , let (L τ , U τ ), (L τ , U τ ), (O τ , I τ ) and (O τ , I τ ) be defined as in Algorithm 10. We also write P τ = G∖(O τ ∪I τ ) and P τ = G ∖ (O τ ∪ I τ ). We only need to prove that, for all τ , (O τ , P τ , I τ ), and (O τ , P τ , I τ ) satisfy Property 1. Since

5 E. 1 .

 51 Appendix E: Remaining proofs for Proposition B.Proofs of the results on DimensionReduction (Appendix D.2) Proof of lemma D.1. It is sufficient to prove that D * cp (h, r) ⊂ Dcp (h, r) ⊂ D * cp (h, r). Recall that we use the convention that

  for illustrations. Then, we define D * W M and D * W M as the population counterparts of DW M with different constants D * W M (T , P , h, r, r cp ) = k ∈ 1, . . . , d ∶ C * (ext) T , P , h, r, r cp ) = k ∈ 1, . . . , d ∶ C * (ext)

  BlockSort with DoubleTrisection -WM Recall the definition (64) of the function Ψ by Ψ(p, r, h, q) = hp √ rq ζ ∧ √ pq + p. In Proposition D.7, we stated a high probability control for the result of BlockSort when fed with DimensionReduction. In particular, this proposition only used the properties of DimensionReduction stated in Lemmas D.1 and D.3. As we have proven in Lemmas F.1 and F.2 (their counterparts for DimensionReduction -WM), we readily obtain the following result whose proof is omitted.

F. 3 .

 3 Analysis of the complete procedure TreeSort with DoubleTrisection -WM In light of Proposition F.3, we need to control the cardinality of Q † W M . In comparison to the oblivious procedure analyzed in the previous section, the main improvement here is that the typical cardinalities Q † W M are smaller than Q † cp thanks to the refined dimension reduction procedure DimensionReduction -WM.Unfortunately, it is not possible to get a tight control of the cardinality of each Q † W M

  3 and define a corresponding subgroup P † , with r † ∈ R, h † ∈ H and a corresponding collection of blocks Q † W M . Define the collection D n = {1, 2, 4, . . . , 2 ⌈log 2 (n)⌉ }. For p ∈ D n , we define P * (p, h, r) as the collection of groups P † satisfying P † ∈ [p, 2p), h † = h, and r † = r.

  Define D * env ∶= D * env (T , P, h, r) = {k ∈ [d] ∶ ∆ * (ext) k,r

i

  ,⌊d l(λ)⌋+1 = 1 and the E ↓(s) ij 's are independent and, for fixed i and j, are i.i.d. distributed and ζ-subGaussian. In fact, the M ↓(s) ij are random since M ↓(s) ij

  3 log c 4 (nd(λ∨1) ζ) , which concludes the proof. by shifting again the right hand-side observation by one. With these simple shifts, ∆(ext) k,rcp and Ĉ(ext) k,2rcp both overestimates ∆ * (ext) k,rcp and C * (ext) k,2rcp and arguing as in the proof of Lemma F.1, we will prove that Q * W M ⊂ QW M with probability at least 1 -δ -see Lemma H.4 below. However, we need to adapt the definition of D * W M (T , P, h, r, r cp ) to cope with this possible bias. Define D * W M -SR-1 (T , P, h, r, r cp ) = k ∈ 1, . . . , d ∶ C * (ext)

2 2 ≤

 2 M i,. -M π(i),.

  If π(j) ≤ i, then M i,. -M j,.

2 2 ≤

 2 M π(j),. -M j,.

2 2 ≤

 2 4l ∞ (π, π * ) .

  λ's. Hence, we can conclude thatinf π sup π * ∈Πn sup M ∶ M π * -1 ∈C BISO E (π * ,M ) [l ∞ (π, π * )] ≥ c ′′ d

E 1 J. 1 . 1 .

 111 [l ∞ (π P C , π * )] ≤ c ′ 1 log c ′ 2 (nd(λ ∨ 1)) turn, implies that E[l ∞ (π P C , π * )] ≤ c log c ′ (nd(λ ∨1)) Proof of the Minimax lower bounds J.1. Proof of Theorem 4.Noiseless minimax lower bound Here, we shall prove the following minimax lower bound holding in the noiseless case ζ = 0. bound remains valid for general ζ ≥ 0. Define the positive integer d -= 1 ∨ [⌊1 λ⌋ ∧ d] ≤ d.

  [n, d, λ, 0] ≥ cd -E[(V -1) + ]. If E[V ] ≥ 2, then we simply use E[(V -1) + ] ≥ E[V ] 2. If E[V ] < 2, we use E[(V -1) + ] ≥ P[V = 2] = n(n-1)2

2 F 1 GĜ( 1 )

 211 , where, in the last line, π(ι) stands for any estimator of the restriction π * to G(ι) . By symmetry, we arrive atR * [n, d,λ, ζ] we have reduced the problem of estimating π * into the sum of n ñ problems of size ñ. Under ν G t(ι) , the restriction of M to G (ι) contains ñ 2 good experts (those in G t(ι) ) and ñ 2 bad experts. The square Euclidean distance between these two types of experts is qυ 2 dζ 2 λ 0d . If we denote Ĝt(ι) the set of the ñ 2 best experts according to π(ι) , then the loss writes asM π(ι)-1 -M π * -∆G t(1) . Since all possible values of G t(1) are ñ 4-apart by definition of the collection G, we deduce that R *[n, d, λ, ζ] ≥

t( 1 )

 1 are block-constant with d blocks of d d questions. Consider the ñ × d matrices N and Y ↓ defined byN i,j = t 1 xt∈{i}×[(j-1)(d d)+1,j(d d)+1] ; Y ↓ i,j = t 1 xt∈{i}×[(j-1)(d d)+1,j(d d)+1] y t -

6 λ ′5 6 ⋀ n 3 4 d 1 4 λ ′3 4 ⋀

 6644 2 that R * [n, d, λ, ζ] ≥ c ′′ nυ 2 ζ 2 ñλ , provided that υ 2 ≤ c ′ λnd ζ 2 ∧n. By choosing υ 2 of the order of the right-hand side, we then deduce that R * [n, d, ζ] ≥ c nζ 2 λ ∧ nd . (156) J.1.4. Step 3. Choice of the parameters and conclusion Writing λ ′ = λ ζ 2 , recall that we aim at proving that R[n, d, λ, ζ] ≥ c nd 1

Case 2 :

 2 Rate n 3 4 d 1 4 λ ′3 4 . This rate can only occur if λ ≥ [1 ∧ (nλ ′3 d)1 4 ] and (a) either n ≤ d and λ′ ∈ [ n d , n 3 d ] or (b) n ∈ [d; d 2 ] and λ ′ ∈ [ n d , d 3 n ].

Case 3 :

 3 Rate n 2 3 √ d λ ′5 6 . This rate can only occur if λ ≥ [1 ∧ λ ′5 6 n 1 3 √ d ] and (a) either n ∈ [d, d 2 ] and λ ′ ∈ [ d 3 n , n 2 ] or (b) n ≥ d 2 and λ ′ ∈ [ n 2 d 3 , n 2 ].

Case 4 :

 4 Rate n √ dλ ′ . This rate can only occur if λ ≥ 1 and λ ′ ≥ (n ∨ d)2 . In this case, we take ñ = 2, d = d, and q = ⌊ √ d⌋. One readily checks that the conditions (153) and (154) are satisfied for a universal numerical value of υ. Then, Proposition J.3 leads to the desired rate. This concludes the proof.J.1.5. Proof of Lemma J.2Proof of Lemma J.2. In order to bound the Kullback-Leibler discrepancy KL(P G P 0 ), we first observe that the rows of N and Y ↓ outside G have the same distribution on P G and P 0 . Besides, all the rows of N and Y ↓ in G are identically distributed on P G and on P 0 . Define the vectors N and Y ↓ by N j = ζ -1 ∑ i∈G N i,j and Y ↓ j = ζ -1

u 2 x

 2 = exp(λ 0 (e u 2 -1)) ∶= exp(I) .

3 28 λ 2 -

 3282 ≤ nde -4 log κ (nd ζ-) ≤ ζ nd e -4 log κ (nd ζ-)+2 log(nd ζ-)

  c 0 Υ where Υ is the sample size. Since the minimax risk is non-decreasing with respect to n, d, and ζ, we only have to consider the case n = 2, d = 1, ζ = 2 c ′′ . Define a = ζ √ Υ. Consider a problem where either M = (a, 0) T or M = (0, a) T . Then, with positive probability, no test is able to distinguish both hypotheses and the risk of any estimator is at most of the order a 2 = ζ 2 Υ. The result follows.

  and h for h † . Since Q * cp corresponds to a set of blocks of questions of size r, it follows that Q * cp ≤ d r. This, in turn, implies that

	√ rh P	Q * cp ≤ P h	√ d and	* P Q cp ≤
	P d. We have proven that

  ). By definition of r min , we have both C * (ext) k,2rcp ≥ h 8 and C * (ext) k,rcp < h 8. Since k ∈ D * cp (P, h, r) and r cp ≥ 8r, we also deduce by monotonocity that the CUSUM of the mean expert in P at scale r cp is higher than h 2, this is C * k,rcp ≥ C * k,8r ≥ h 2 since k ∈ D * cp (P, h, r) -see (55). Remark that, since r cp ≤ r, we have

  -SR-2 (P , h, r, r cp ). Since r cp takes at most a logarithmic number of values and since P

1 hrcp = c ′ (hr) regions of Q r that contain at least a point D * W M * (p) ≤ n p, we obtain rcp P ∈P * (p)

In the Poissonian scheme, each entry is observed at least once with probability 1e -λ .

This would correspond to the situation where the corresponding ordering of the questions is also unknown.

Appendix D: Proof of Proposition B.5

In this section, we prove Proposition B.5 and thereby control the loss of the estimator π with simple dimension reduction. For this purpose, we analyze each step of the algorithm. In Appendix D.2, we first prove that, by detecting the high-variation regions of M , we are able to aggregate M at some scale r without decreasing much the variation of M . This allows us to drastically reduce the dimension of the problem. Then, in Sections D.3 and D.3, we show that, unless this aggregated matrix Θ has small variations, DoubleTrisection -PCA and Pivot will remove some experts so that the corresponding new aggregated matrix Θ ′ exhibit significantly smaller variations. As a consequence, after a polylogarithmic number of iterations of the procedure, the variations of the matrix M restricted to the remaining experts of P is small enough.

Appendix H: Permutation estimation in the semi-random model

H.1. Model and Algorithm

We now consider a slightly different model with Υ * samples Y (1) , . . . , Y (Υ * -1) . The noise matrices E (1) , . . . , E (Υ * -1) are sampled independently (as previously) and

]. This slightly different model is mainly motivated by the analysis of the partial observation scheme in Section 4. In particular, building upon this model and relying on the corresponding modifications in the algorithm allows us to recover the right dependency with respect to ζ in Section 4.

We consider a slight variant πW M -SR of the estimator πW M to handle the adversarial differences. The procedure πW M -SR is computed exactly as πW M except that • In Pivot (Algorithm 4), the threshold β tris log(

Theorem H.1. There exist three numerical constants c, c ′ , and c 0 such that the following holds. Fix δ > 0 and assume that Υ ≥ c 0 log 8 (nd (δζ -)). For any permutation π * ∈ Π n and any matrix M such that M π * -1 ∈ C BISO , the hierarchical sorting tree estimator with memory πW M -SR satisfies

with probability at least 1 -c ′ n log 9 ( nd δζ-)δ.

H.2. Proof of Theorem H.1

The proof follows the main steps as that of Theorem 2.3 and we mainly emphasize here the differences. In the proof of Theorem 2.3, we often work with the aggregated model (50) Z = Θ + N which is restricted to a subset P of experts and a subset Q ⊂ Q r of questions aggregated at scale r -see Encode -Matrix for details. For t = 0, . . . , Υ -1, the counterpart of (50) is the following

where the entries of N (t) are independent and ζ-subGaussian and Θ (t) stands for the corresponding aggregation of the matrix M (t) . Since the total variation of each row of M is at most one, one readily checks that j∈Q

Since πW M -SR is a hierarchical sorting tree estimator, we are in position to control its loss using Proposition B.1. For this purpose, we need to prove that Proposition B.3 still holds in the semi-random model which, in turn, would imply that Corollary B.4 is true. In fact, the proof of Proposition B.3 is verbatim the same except that Lemma C.1 is replaced by the following lemma.

We remind that P ′ = P ∖ (L ∪ U ), and

Lemma H.2. For any non-zero vector w ∈ R Q + , any pivot γ ∈ {1, . . . , P } , we have P[P 3 ] ≥ 1 -δ. Besides, on the same event of probability at least 1 -δ, we have

Proof of Lemma H.2. Consider any sample t ∈ [0, Υ -1], any vector w ∈ R q , and any i ∈ P . As a straightforward consequence of (121), we deduce that

We then deduce from a union bound, that with probability higher than 1 -δ, we have

simultaneously for all i in P . The rest of the proof of Lemma H.2 is left unchanged provided that we replace 2 log

Then, being in position to apply Corollary B.4, we state the counterpart of Proposition B.6.

Proposition H.3. On the intersection of event ξ (defined in Corollary B.4) and an event of probability higher than 1 -5 ⋅ 2 t τ ∞ δ, it holds that

We conclude the proof of Theorem H.1 by combining Proposition H.3 with Corollary B.4. Hence, we only need to prove the last proposition.

H.3. Proof of Proposition H.3

Again, we only emphasize the differences with the proof of Proposition B.6. We start with the analysis of DimensionReduction -WM. Recall that we slightly changed the definition of the CUSUM statistics

by shifting the second sum by one index. The definition of the population CUSUM statistic

k,2rcp is left unchanged. Similarly, we slightly changed the definition of

The control of ∑ l∈S * c (z * l ) 2 is slightly different.

i,l -θ

where we used (136) as well as the fact Θ

). Combining Appendix H.4, (140), and Condition (130), we deduce that

which, together with (139), yields

)

)

Then, we come back to the matrix Θ -Θ using again (121).

)

Then, we apply Harris' inequality as in the original proof of the lemma to conclude that

Applying the pivot algorithm to ŵ+ , we deduce from Lemma H.2 that there exists an event of probability higher than 1 -δ such that max i,j∈P

By convexity, it follows that

+ 800 .

J.1.3. Construction of the Prior distribution on M

Let ñ ∈ [n] be an a power of 2 so that n ñ is an integer. From a broad perspective, the general purpose of this prior construction is to break down the permutation estimation problem into n ñ independent bisection problems of size ñ. We will fix the value of ñ at the end of the proof. The permuted matrix M π * -1 will turn out to be block constant and we introduce d ∈ [d] the number of blocks of questions, each of them being of size d d. Here we assume that d is a power of 2 so that d d is an integer. d will be also fixed at the end of the proof. We introduce the staircase matrix C of dimension (n ñ)× d such that C ι,κ = ιñ (4n)+κ (4 d). Also write U for the constant ñ × d d matrix whose entries are all equal to one. With this notation, the Kronecker product matrix C ⊗ U of size n × d is a bi-isotonic staircase matrix with blocks of size ñ × (d d).

Then, we shall perturb the matrix C ⊗ U in order to simultaneously craft n ñ independent clustering problems of size ñ each. Set λ = λ d d and λ 0 = ñλ . Let υ be a positive number and let also q be an integer smaller than or equal to d and

where the random matrix B (full) ∈ {0, 1} n×d is defined below.

For this purpose, we consider a collection G of subsets of [ñ] with size ñ 2 that are wellseparated in symmetric difference as defined by the following lemma.

Lemma J.1. There exists a numerical constant c 0 such that the following holds for any even integer ñ. There exists a collection G of subsets of [ñ] with size ñ 2 whose satisfies log( G ) ≥ c 0 ñ and whose elements are ñ 4-separated, that is

The above result is a straightforward consequence of Varshamov-Gilbert's lemma -see e.g. [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF].

For each block ι ∈ [n ñ], we fix a subset G (ι) from G. Then, we consider its 'translation' G t(ι) = {x + (ι -1)ñ ∶ x ∈ G ι }. The experts of G t(ι) will correspond to the subgroup of 'higher' experts in the group ι. We write G = (G t(1) , . . . , G t(n ñ) ) and G the corresponding collection of all possible G. Given any such G, we shall define a prior distribution ν G on M .

For ι ∈ [n ñ], we sample uniformly a subset Q (ι) of q block of questions among the d blocks. In each of these q blocks, the corresponding rows of B (full) are equal to one. More formally, upon writing 1 d d for the constant vector of size d d, we have

To sum up, we define a prior distribution ν G on B (full) (and equivalently on M ) such that, under ν G , all the rows of B (full) that do not belong to any G t(ι) are zero. All the rows belonging to the same set G t(ι) are equal and block constants with d blocks of size d d, among which q blocks are exactly equal to one.

Coming back to the matrix M defined in (148), we see that as soon as

then, almost surely, the matrix M , is up to a (non-unique) permutation, bi-isotonic and its

Experts in group ι, not in G (ι) Experts in groups ι-1 and ι+1

Questions corres. to (Q ν G , recovering a suitable permutation π * is exactly equivalent to estimating the subgroup G t(ι) ⊂ G (ι) for each ι = 1, . . . , n ñ. This construction of M is illustrated in Figure 6. To sum up, the prior distribution distribution ν G on M requires the choice of the parameters ñ ∈

and some signal level υ > 0 satisfying (149).

As we shall use several reduction arguments, we need to introduce some new notation. First, we respectively denote P (full) G and E (full) G for the marginal probability and expectation with respect to the data when M is sampled according to ν G .

The distribution of the rows G t(ι) in M under ν G only depends on G t (ι) . In what follows, we write ν G t(ι) for this distribution. Similarly, we write P (full) G t(ι) for the corresponding marginal distribution of the observations (x t , y t ) such that (x t ) 1 ∈ G t(ι) . By the poissonization trick, the distribution P (full) G is a product measure of P (full) G t(ι) for ι = 1, . . . , n ñ. We write E (full) G t(ι) for the corresponding expectation.

Step 2: Problem Reduction We start with prior distributions ν G .

For each of these matrices M sampled from a distribution ν G , it turns out that π * (G

Hence, to estimate π * , we only need to estimate each G t(ι) ⊂ G (ι) from the data. Intuitively, we therefore can restrict ourselves to estimators π satisfying π(G

) = G (ι) . More precisely, if an estimator π does not satisfy this condition, then we can modify π in π in order to enforce the G (ι) 's to be be stable. Since, by Condition (149) experts in different G (ι) are far from each