Dr Hab Bartosz Kołodziejek

Dr Michał Bosy

Dr Hab Ulrike Schneider

Dr Clément Elvira

Dr Hab Cédric Herzet

Dr Hab Wojciech Rejchel

Dr Tomasz Stroiński

Dr Samuel Vaiter

Prof Piotr

Zwiernik for mind-opening discussions, which helped me seeing the other branches of the currently presented research. I would also want to thank the anonymous referees of our articles for their insightful comments and for improving them. This dissertation would be not possible to complete without a good atmosphere in the academic community. I would like to thank my colleagues from 6.06 and KNM for our long discussions on mathematics, life et al. Special thanks are due to prof. Janusz Górniak for his enormous help during the beginning of my academic adventure and to Monika Kaczmarz for her invaluable

Streszczenie

Niniejsza rozprawa poświęcona jest nowym zastosowaniom geometrii dyskretnej i kombinatoryki w nowoczesnej statystyce. Pierwsze z nich skupione jest na jednym z popularniejszych rozwiązań na radzenie z ciągłym przyrostem danych, jest nim penalizowana regresja liniowa. Mając na celu odtworzenie potrzebnych nam własności wektora współczynników regresji, rozpoczynamy dyskusję od estymatora SLOPE (Sorted ℓ 1 Penalized Estimator), który został wprowadzony w poprzedniej dekadzie. Szczególną uwagę poświęcamy pojęciu wzorca SLOPE, który zachowuje informację o nośniku, znaku i rankingu między współczynnikami regresji. Informuje on również o klastrach współczynników o tej samej wartości bezwzględnej. W rozdziale trzecim podajemy warunki, dla których SLOPE poprawnie odtwarza nośnik oraz klastry wektora współczynników regresji przy ortogonalnej macierzy eksperymentu. Przy tym założeniu wyprowadzamy też nowe wyniki dotyczące mocnej zgodności estymatora SLOPE i jego wzorca. Rozdział czwarty rozszerza dyskusję na temat SLOPE, pomijając założenie o ortogonalności macierzy eksperymentu. Wprowadzamy warunek niereprezentowalności dla SLOPE, który jest konieczny i dostateczny do odtworzenia wzorca w przypadku braku szumu, po czym ilustrujemy ten warunek geometrycznie. Następnie rozważamy przypadek asymptotycznego przyrostu liczby zmiennych objaśniających i szumu rosnącego inkrementalnie. W rozdziale piątym omawiamy szerszą klasę penalizowanych estymatorów zwaną polyhedral gauges. Pozwala ona na wykorzystanie twierdzeń z geometrii wielościanów do uogólnienia pojęcia wzorca i wyników dotyczących jego odtwarzania. Rozdział szósty dotyczy istnienia estymatora największej wiarogodności (MLE) w dyskretnych rodzinach wykładniczych. Podajemy jego pełną charakteryzację za pomocą pojęcia zbioru jednoznaczności. Następnie badamy rozmiar próby niezależnych zmiennych losowych o tym samym rozkładzie, która zapewnia istnienie MLE z wysokim prawdopodobieństwem. W tym celu wykorzystujemy narzędzia z analizy hipersześcianów dyskretnych i stosujemy otrzymane wyniki w modelach wykładniczych grafów losowych. Ostatni z rozdziałów skupiony jest na połączeniu między teorią modeli graficznych w statystyce, a pojęciami laplasjanu grafu oraz dyskretyzacji procesów Wienera. Rozprawa jest oparta na trzech opublikowanych artykułach oraz dwóch preprintach dostępnych on-line.

x CHAPTER 0. STRESZCZENIE (SUMMARY IN POLISH)

Résumé

Cette thèse traite des applications de la combinatoire et de la géométrie discrète aux statistiques modernes. La première application porte sur l'une des approches fréquemment utilisées pour faire face au volume croissant des données, à savoir l'utilisation de méthodes de régression linéaire pénalisée. Dans le but de retrouver la structure que possède le vecteur des coefficients de régression, nous commençons notre discussion par l'estimateur SLOPE (Sorted ℓ 1 Penalized Estimator), qui a été proposé il y a presque dix ans. En particulier, nous examinons la notion de schéma SLOPE, qui conserve l'information sur le support, le signe et le classement des coefficients de régression. En particulier, il préserve les groupes de coefficients ayant la même valeur absolue. Dans le chapitre 3, nous fournissons les conditions, sous lesquelles SLOPE récupère l'ensemble des variables pertinentes et des groupes lorsque la matrice de planification est orthogonale. Nous déduisons également de nouveaux résultats sur la forte consistance de l'estimateur SLOPE et de son schéma. Le chapitre 4 étend la discussion sur l'estimateur SLOPE à une classe générale de matrices de planification fixes. Nous fournissons la condition d'irreprésentabilité du SLOPE, qui est nécessaire et suffisante pour la recouvrement du schéma dans le cadre non-bruité et nous illustrons cette propriété géométriquement. Dans un deuxième temps, nous considérons le cas asymptotique lorsque le nombre de variables explicatives tend vers l'infini et que l'erreur est incrémentale. Dans le chapitre 5, nous étudions la classe plus large des estimateurs pénalisés, appelés les jauges polyédriques. Elle permet d'utiliser les notions issues de la géométrie des polyèdres pour généraliser la notion du schéma et les résultats sur sa récupération. Le chapitre 6 s'articule autour de l'existence de l'estimateur du maximum de vraisemblance (MLE) pour les familles exponentielles discrètes. Nous donnons sa nouvelle caractérisation basée sur la notion d'ensemble d'unicité. Par la suite, nous inspectons la taille des échantillons indépendants identiquement distribués qui est nécessaire pour assurer son existence avec une grande probabilité. Pour cela, nous utilisons les notions issues de l'analyse des hypercubes discrets et appliquons nos résultats dans l'environnement des graphes aléatoires. Le dernier des chapitres relie la théorie des schémas graphiques en statistique avec la notion de matrices laplaciennes de graphes et de processus de Wiener discrétisés. La thèse est basée sur trois articles déjà publiés et deux prépublications, qui sont disponibles en ligne.

Wstęp

Obecnie można zauważyć szybki i nieunikniony przyrost danych, zarówno pod względem liczby obserwacji, jak i pod względem liczby sposobów, za pomocą których da się je zmierzyć. Duże zbiory danych i ich analiza rosną na znaczeniu w życiu codziennym, dzięki czemu statystyka matematyczna oraz nauki związane z analizą danych zyskują coraz większe zainteresowanie w innych działach matematyki i jej zastosowaniach. Jednak statystyka i analiza danych nie powinny być traktowane w oderwaniu od innych działów matematyki. Kluczowym punktem badań niniejszej rozprawy będzie znajdowanie nowych połączeń między współczesną statystyką, geometrią i kombinatoryką. Dyskusja podjęta w tej rozprawie rozpoczyna się od rozważania przestrzeni euklidesowych o skończonym wymiarze. Zwracamy uwagę na przypadki, w których zbiór wartości danego estymatora można podzielić na skończoną liczbę podzbiorów w taki sposób, żeby istniała bijekcja między nami, a interesującymi nas własnościami tego estymatora. Taki sposób predykcji obserwacji jest znany jako problem klasyfikacji [START_REF] Boucheron | Theory of classification: a survey of some recent advances[END_REF], nad którym badania zostały rozpoczęte przez Fishera [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF] i który jest stosowany w prawie każdym dziale nauk związanych z przetwarzaniem danych.

PL.1 Penalizowana regresja liniowa

Za jedno z istotnych zastosowań problemu klasyfikacji możemy uznać wersję regresji liniowej, w której zamiast dokładnego oszacowania wartości nieznanego wektora interesują nas jego wybrane właściwości. W modelu regresji liniowej wielorakiej mającym n obserwacji i p zmiennych objaśniających zakładamy, że zmienna objaśniana Y = (Y 1 , . . . , Y n ) ′ ∈ R n jest postaci Y = Xβ + ε, gdzie X ∈ R n×p jest macierzą eksperymentu, β = (β 1 , . . . , β p ) ′ ∈ R p jest nieznanym wektorem współczynników regresji, a ε = (ε 1 , . . . , ε n ) ′ ∈ R n jest losowym wektorem błędu (szumu). Głównym zagadnieniem regresji liniowej są oszacowanie β oraz wydobycie jego istotnych własności. Szeroka klasa zagadnień opartych na wyborze podzbioru współrzędnych β, które będą najlepiej spełniać oczekiwania dotyczące estymatora i jego złożoności, znany jest jako problem wyboru modelu. Więcej informacji o tym zagadnieniu można znaleźć w przeglądowym artykule [START_REF] Claeskens | Statistical model choice[END_REF]. Przez większość dyskusji zakładamy, że wektor błędu ε jest losowy, a jego rozkład jest ciągły i symetrycznyn. Niektóre z przedstawionych wyników dotyczą również przypadku niezaszumionego, w którym ε ≡ 0, co pomoże w wyprowadzeniu wyników asymptotycznych w przypadku zaszumionym. Dzięki istniejącym wynikom dotyczącym rozkładu normalnego możemy pokazać ulepszone wyniki dla szumu gaussowskiego ε ∼ N (0, σ 2 I n ). Najbardziej klasyczna metoda regresji liniowej, metoda najmniejszych kwadratów (Ordinary Least Squares, OLS), została wprowadzona przez Legendre'a na początku XIX wieku [START_REF] Barbin | Histoires de probabilités et de statistiques[END_REF][START_REF] Hald | A history of parametric statistical inference from Bernoulli to Fisher, 1713-1935[END_REF][START_REF] Legendre | Nouvelles méthodes pour la détermination des orbites des comètes. Nineteenth Century Collections Online[END_REF] Popularność estymatora najmniejszych kwadratów ma miejsce ze względu na swoją prostotę, dokładność przy względnie małej liczbie zmiennych [33, str. 4] i wielu innych przydatnych statystycznych własności. Dla przykładu, jeżeli macierz X ′ X jest odwracalna, a Cov(Y ) = σ 2 I n dla σ > 0, to OLS jest najlepszym estymatorem w klasie nieobciążonych liniowych estymatorów (BLUE) [1, Theorem 2.7.1.] wektora β. W tym przypadku istnieje wzór jawny na βOLS , który można w dosyć łatwy sposób wyprowadzić [1, str. 28]: βOLS = (X ′ X) -1 X ′ Y .

Po dodaniu założenia o gaussowskości wektora Y estymator OLS staje się estymatorem największej wiarogodności (MLE) [1, str. 28]. W przypadku, gdy macierz eksperymentu X jest ortogonalna, tzn. X ′ X = cI p , c > 0, powyższy wzór sprowadza się do βOLS = 1 c X ′ Y . Przy powyższych założeniach OLS jest również mocno zgodnym estymatorem dla β [START_REF] Anderson | Strong consistency of least squares estimates in normal linear regression[END_REF]. Z drugiej strony, estymator ten nie jest jednoznacznie zdefiniowany, kiedy macierz X ′ X jest nieodwracalna, co ma miejsce w wysokowymiarowym przypadku, gdy p > n. Ponadto, przy ogólnie przyjmowanych założeniach o błędzie ε, OLS nie jest estymatorem rzadkim, z prawdopodobieństwem 1 składa się on z p współrzędnych o parami różnych wartościach. Mniej klasyczne założenia o wektorze Y i macierzy X, przy których estymator nie jest jednoznaczny, można znaleźć między innymi w niedawno opublikowanym artykule Dupuisa i Vaitera [START_REF] Dupuis | The geometry of sparse analysis regularization[END_REF]. W praktyce często wektor β składa się ze względnie małej liczby niezerowych współrzędnych, co w naturalny sposób sugeruje metody, które promują rzadkość wektora β rozumianą jako małą liczbę niezerowych współrzędnych lub opisywalność β za pomocą małej liczby parametrów. Istnieją różne propozycje rozwiązania powyższego problemu, m.in. porównanie pasujących modeli przy pomocy kryterium informacyjnego, np. BIC [START_REF] Schwarz | Estimating the dimension of a model[END_REF] lub AIC [START_REF] Akaike | A new look at the statistical model identification[END_REF]. Innym podejściem jest penalizowana regresja liniowa postaci: β := arg min b ∈R p 1 2 ∥Y -Xb∥ 2 2 + pen λ (b) , gdzie pen λ (b) jest ustaloną nieujemną karą, którą można modyfikować za pomocą parametru λ > 0. To podejście jest wykorzystywane m.in. w neuroobrazowaniu [START_REF] Brzyski | Connectivity-informed adaptive regularization for generalized outcomes[END_REF], prognozowaniu cen energii elektrycznej [START_REF] Jedrzejewski | Electricity price forecasting: The dawn of machine learning[END_REF][START_REF] Uniejewski | Automated variable selection and shrinkage for day-ahead electricity price forecasting[END_REF] oraz w matematyce finansowej w celu grupowaniu aktywów na podstawie ich korelacji częściowej z szeregiem czasowym stóp zwrotu z funduszy inwestycyjnych [START_REF] Kremer | Sparse index clones via the sorted ℓ 1 -norm[END_REF]. Pierwszym z zaproponowanych sposobów penalizowanej regresji była metoda wyboru najlepszego podzbioru [START_REF] Beale | The discarding of variables in multivariate analysis[END_REF][START_REF] Hocking | Selection of the best subset in regression analysis[END_REF], gdzie kara pen λ (b) = λ∥b∥ 0 jest równa liczbie współrzędnych niezerowych wektora b pomnożonej przez λ. Kłopot z jej zastosowaniem polega na tym, że dla dużych wartości p znalezienie najlepszego podzbioru w ten sposób jest problemem NP-trudnym [START_REF] Freijeiro-González | A critical review of LASSO and its derivatives for variable selection under dependence among covariates[END_REF]. Inną metodą na znalezienie rzadkiego rozwiązania jest metoda LASSO (skrót od Least Absolute Shrinkage and Selection Operator [START_REF] Chen | Basis pursuit[END_REF][START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF]), w której karą nałożoną na estymator b jest jego norma ℓ 1 pomnożona przez parametr λ > 0:

βLASSO := arg min b ∈R p 1 2 ∥Y -Xb∥ 2 2 + λ∥b∥ 1 .
Estymator LASSO jest obciążony. Jako estymator ściągający (shrinkage estimator) sprowadza on współrzędne βLASSO w stronę zera. Niektóre z nich zostają ściągnięte całkowicie do zera, co przynosi rezultat w postaci rzadszego estymatora. W przypadku, gdy macierz eksperymentu X jest ortonormalna, tzn. X ′ X = I p , jawny wzór na βLASSO , wyprowadzony w pracy Tibshiraniego [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF], jest oparty na βOLS :

βLASSO i = sign( βOLS i ) max | βOLS i | -λ, 0 .

PL.2. SLOPE xv

Aby zapewnić istnienie rozwiązania zagadnienia penalizowanej regresji liniowej, często używana jest kara wypukła, co umożliwia wykorzystanie narzędzi z analizy wypukłej. Zainteresowanym różnymi przykładami pomysłów na modyfikacje estymatora LASSO można polecić artykuł [START_REF] Freijeiro-González | A critical review of LASSO and its derivatives for variable selection under dependence among covariates[END_REF]. Warto zauważyć, że kary niewypukłe również są wykorzystywane m.in. przy wyborze najlepszego pozdbioru [START_REF] Beale | The discarding of variables in multivariate analysis[END_REF][START_REF] Hocking | Selection of the best subset in regression analysis[END_REF] lub estymatora SCAD (smoothly clipped absolute deviation [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF]).

PL.2 SLOPE

Innym sposobem na zmniejszenie wymiaru jest estymator SLOPE (Sorted ℓ 1 Penalized Estimator [START_REF] Bogdan | Statistical estimation and testing via the sorted l1 norm[END_REF][START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF][START_REF] Zeng | Decreasing weighted sorted ℓ 1 regularization[END_REF]), który poza uogólnieniem metody LASSO, skleja takie same współczynniki regresji β oraz skorelowane kolumny macierzy X. Zgodnie z nazwą, w estymatorze SLOPE norma ℓ 1 w funkcji kary została zastąpiona przez zdefiniowaną poniżej posortowaną normę ℓ 1 :

J Λ (b) := p i=1 |b| (i) λ i ,
gdzie λ 1 > 0, λ 1 ≥ . . . , λ p ≥ 0, natomiast |b| (1) ≥ . . . ≥ |b| (p) to wartości bezwzględne współrzędnych wektora b posortowane malejąco. Szczególny przypadek SLOPE z Λ będącym ciągiem arytmetycznym jest znany pod nazwą OSCAR [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF]. Zauważmy, że w parametr λ został zastąpiony w SLOPE przez nierosnący wektor Λ = (λ 1 , λ 2 , . . . , λ p ) ′ parametrów (tuning vector). Możemy więc nie tylko modyfikować wielkość wektora Λ, ale również jego kształt. Główną motywacją autorów metody SLOPE było testowanie p hipotez zerowych H 0 i : β i = 0 i kontrola współczynnika fałszywych odkryć (FDR control) zdefiniowanego jako oczekiwana proporcja między liczbą fałszywych odrzuceń hipotezy zerowej, a łączną liczbą jej odrzuceń (w przypadku braku odrzuceń przyjmuje się FDR = 0). Ponadto, SLOPE jest uogólnieniem poprzednich metod:

• λ 1 = . . . = λ p = 0 ⇒ βSLOPE = βOLS ,

• λ 1 = . . . = λ p > 0 ⇒ βSLOPE = βLASSO ,

• Λ is an arithmetic sequence ⇒ βSLOPE = βOSCAR . Dokładniejszy opis historii badań nad estymatorem SLOPE znajduje się w podrozdziale 4.1.1. W naszych badaniach skupiamy się na odtworzeniu istotnych właściwości wektora β za pomocą estymatora SLOPE, które są zakodowane w wektorze zwanym wzorcem SLOPE. Tak jak znak wektora β w przypadku LASSO, wzorzec SLOPE można całkowicie opisać za pomocą subróżniczki funkcji kary. Dokładniej mówiąc, niech k będzie liczbą klastrów wektora β, tzn. liczbą niezerowych różnych wartości współrzędnych |β|. W rozdziale trzecim i czwartym omawiamy nowe warunki konieczne i dostateczne na odtwarzanie wzorca SLOPE, jak również nowe wyniki o zgodności i mocnej zgodności SLOPE i jego wzorca w przypadku, gdy n ≥ p.

PL.2.1 Zgodność

Głównym narzędziem do udowodnienia mocnej zgodności SLOPE w przypadku n ≥ p jest mocna zgodność estymatora największych kwadratów udowodniona m.in. w pracy Andersona i Taylora [START_REF] Anderson | Strong consistency of least squares estimates in normal linear regression[END_REF]. Główne wyniki dotyczące zgodności LASSO można znaleźć w następujących publikacjach:

• zgodność LASSO: Knight, Fu, 2000 [START_REF] Knight | Asymptotics of Lasso-type estimators[END_REF],

• mocna zgodność LASSO: Chatterjee, Lahiri, 2011 [START_REF] Chatterjee | Strong consistency of Lasso estimators[END_REF],

Główne założenia: λ n jest rzędu mniejszego niż n oraz E|ε| < ∞.

• zgodność znaku LASSO: Zhao, Yu, 2006 [START_REF] Zhao | On model selection consistency of Lasso[END_REF].

Podrozdziały 3.4 oraz 4.6 niniejszej rozprawy dotyczą mocnej zgodności SLOPE i jego wzorca. Warto zauważyć, że w przypadku, gdy wektor Λ jest stały (LASSO), wzorzec SLOPE nie jest zgodny, nawet jeśli założenia o zgodności wektora znaku są spełnione. Estymator LASSO nie odtwarza klastrów, które są elementem wzorca SLOPE. Zauważmy też, że mocna zgodność estymatora LASSO (SLOPE) nie implikuje mocnej zgodności jego znaku (ani jego wzorca). Jako kontrprzykład można rozważyć ciąg ((1/n, 1/n, -1/n, -1/n) ′ ) n≥1 , który zbiega do (0, 0, 0, 0) ′ , podczas gdy jego znak i jego wzorzec SLOPE są równe (1, 1, -1, -1) ′ dla wszystkich n.

PL.2.2 Warunek niereprezentowalności

Dla danych n, p > 0 zgodność zbioru niezerowych współrzędnych wektora βLASSO jest prawie równoważna do warunku, w którym współrzędne spoza nośnika wektora β nie są reprezentowane przez współrzędne doń należące [START_REF] Zhao | On model selection consistency of Lasso[END_REF]. Z tego powodu warunek został nazwany warunkiem niereprezentowalności (irrepresentability condition). W rozdziale 4. rozprawy wprowadzony jest analogon powyższego warunku dla estymatora SLOPE.

PL.2. SLOPE

xvii

β 1 β 2 SLOPE OLS SLOPE OLS SLOPE OLS (-2,1) (-1,2) (1,2)
(2,1)

(2,-1)

(1,-2) (-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1)
(1,0)

(1,-1) (0,-1) (-1,-1) (0,0) βSLOPE and βOLS w przypadku ortogonalnym: X ′ X = I p for Λ = (2, 1) ′ .

PL.2.3 Penalizowana regresja liniowa i jej geometria

Dla lepszego zrozumienia powiązań między βSLOPE , a βOLS , ilustrujemy je w przypadku niskowymiarowym p ≤ n dla p = 2. Tutaj wartości estymatora SLOPE jest równa różnicy między estymatorem najmniejszych kwadratów, a jednym z jego rzutów na (X ′ X) -1 C Λ , gdzie C Λ jest kulą jednostkową w normie dualnej do J Λ . Powyższy wynik ma łatwą interpretację w przypadku ortogonalnym, w którym zarówno X, jak i rzut na C Λ , są ortogonalne. Ta zależność ma rezultat w postaci jawnego wzoru na βSLOP E w przypadku ortogonalnym, który został niedawno wprowadzony przez Tardivela, Serviena i Concordeta [START_REF] Tardivel | Simple expressions of the lasso and slope estimators in low-dimension[END_REF]. Więcej ilustracji związków między estymatorem SLOPE, a jego geometrią, można znaleźć w podrozdziale 4.4. Zachodzi połączenie między estymatorami LASSO i SLOPE, a systemami pierwiastkowymi (root systems) wykorzystywanymi w analizie harmonicznej. Dokładniej mówiąc, można zauważyć, że kula jednostkowa w normie ℓ ∞ dualnej do kary stosowanej w LASSO oraz kula jednostkowa w normie J * Λ dualnej do normy SLOPE są proporcjonalne odpowiednio do powłok wypukłych orbit grup Weyla odpowiadających p-tej potędze systemu pierwiastkowego A 1 oraz systemowi B p [START_REF] Godland | Projections and angle sums of permutohedra[END_REF][START_REF] Postnikov | Permutohedra, associahedra, and beyond[END_REF]. Więcej na temat systemów pierwiastkowych można znaleźć m.in. w książce Helgasona [START_REF] Helgason | Differential geometry, Lie groups, and symmetric spaces[END_REF] oraz w publikacjach [START_REF] Graczyk | Potential kernels for radial Dunkl Laplacians[END_REF][START_REF] Dziubański | Remark on atomic decompositions for the Hardy space H 1 in the rational Dunkl setting[END_REF]. Geometrię penalizowanej regresji liniowej i odtwarzania wzorców jej estymatorów można rozważać w ogólniejszym przypadku, co robimy w rozdziale piątym. Mianowicie, estymator SLOPE można zaklasyfikować do jednej ze skończonej liczby klas, gdy kara jest postaci polyhedral gauge, tzn. nieujemną funkcją wypukłą równą maksimum skończonej liczby funkcji liniowych. Istotne właściwości estymatora, również nazwane jego wzorcem (pattern), można całkowicie opisać za pomocą subróżniczki kary. Ponieważ nierówności liniowe definiujące polyhedral gauge są spełnione przez przekrój skończonej liczby związanych z nimi półprzestrzeni, zachodzi tutaj silne powiązanie z teorią wielościanów, z którą można zapoznać się. m.in. w książkach Grubera [START_REF] Gruber | Convex and Discrete Geometry[END_REF], Grünbauma [START_REF] Grünbaum | Convex polytopes[END_REF], Hiriart-Urrutiego i Lemarechala [START_REF] Hiriart-Urruty | Convex Analysis and Minimization Algorithms I: Fundamentals[END_REF] oraz Zieglera [START_REF] Ziegler | Lectures on Polytopes[END_REF]. W rozdziale piątym uogólniamy niektóre z nowo otrzymanych wyników SLOPE na powyższą klasę estymatorów. W tym celu rozważamy pojęcie klasy równoważności wzorca (pattern equivalence class), która pozwala nam opisać odtworzenie wzorca jako odtworzenie wartości sub-

β 1 β 2 OLS SLOPE (1,2) (2,1) (-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1)
(1,0)

(1,-1) (0,-1) (-1,-1) (0,0) βSLOPE oraz βOLS dla X ′ X = 1 0.6 0.6 1 and Λ = (2, 1) ′ . różniczki kary pen. Wprowadzamy i charakteryzujemy pojęcie osiągalności (accessibility) wzorca i wprowadzamy warunek niereprezentowalności dla polyhedral gauges. Z pomocą narzędzi z geometrii wielościanów, w szczególności pojęcia stożków normalnych ścian rozważanych wielościanów, rozważamy własności równoważności wzorca. Następnie skupiamy się na odtwarzaniu modelu przez estymatory progowane. Są one uogólnieniem progowanego LASSO, jednak w odróżnieniu od swojego poprzednika ich celem jest zmniejszenie złożoności estymatora nie tylko poprzez sprowadzenie części współrzędnych do zera, ale również do innych jego uproszczeń związanych z danym wzorcem. Rozszerzamy też pojęcie jednorodnej jednoznaczności estymatora znanej dla norm wielościennych [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] do polyhedral gauges.

PL.3 Dyskretne rodziny wykładnicze

W rozdziale szóstym przechodzimy z przestrzeni euklidesowych do przestrzeni ze skończoną liczbą elementów (dyskretnych). Punktem naszych zainteresowań są rodziny wykładnicze zdefiniowane na skończonej przestrzeni stanów X , |X | = K < ∞. Z powiązaną dyskusją na temat przeliczalnych zbiorów można zapoznać się w pracy Jacobsena [START_REF] Jacobsen | Existence and unicity of MLEs in discrete exponential family distributions[END_REF]. My rozważamy podprzestrzeń liniową B przestrzeni funkcji R X , do której należy funkcja stała dodatnia. Na zbiorze X wprowadzamy też ściśle dodatnią wagę µ : X → (0, ∞). Rodzinę wykładniczą konstruujemy następująco: Dla rzeczywistej funkcji ϕ definiujemy odpowiednio jej funkcję partycji oraz log-partycji: Dzięki temu możemy wprowadzić funkcję wiarogodności i log-wiarogodności. Ta ostatnia jest ściśle wklęsła, co zapewnia nam jednoznaczność MLE w przypadku jego istnienia. Istnienie estymatora nie jest zapewnione pomimo faktu, że funkcja wiarogodności jest ograniczona. W naszych badaniach wyprowadzamy nowe kryterium na istnienie MLE i wykorzystujemy je w rodzinach wykładniczych rozpiętych przez funkcje Rademachera i Walsha oraz w wykładniczych modelach grafów losowych. Naszym głównym narzędziem jest pojęcie zbioru jednoznaczności, czyli takiego zbioru U ⊂ X , że ϕ = 0 jest jedyną funkcją z zadanej klasy, która jest równa zero na całym zbiorze U . Dokładniej, pokazujemy, że MLE istnieje dla e(B) oraz próby i.i.d. x 1 , x 2 , . . . , x n ∈ X wtedy i tylko wtedy, gdy zbiór {x 1 , . . . , x n } jest jednoznaczności dla nieujemnego stożka funkcji B + := {ϕ ∈ B : ϕ ≥ 0}. Powyższe kryterium podajemy również jako zagadnienie programowania liniowego. Dodatkowo, w rozważanych zastosowaniach wyznaczamy asymptotyczne wielkości zbiorów jednoznaczności. W tym celu używamy klasycznych wyników z teorii grafów losowych oraz z problemu zbieracza kuponów (Coupon Collector's Problem) [118, str. 194-195], [START_REF] Erdős | On a classical problem of probability theory[END_REF], and [START_REF] Pósfai | Approximation theorems related to the coupon collector's problem[END_REF]. W rodzinie rozpiętej przez funkcje Walsha wykorzystujemy własności hipersześcianu {-1, 1} k (k = log 2 K), jego podkości oraz związanego z nim grafu hipersześcianu.

Z(ϕ) =

PL.4 Modele graficzne

Jednym z działów nowoczesnej statystyki wykorzystujących zarówno MLE, jak i estymatorów penalizowanych jest teoria modeli graficznych (graphical models). Model graficzny jest rodziną rozkładów prawdopodobieństwa skończonego zbioru zmiennych losowych X 1 , X 2 , . . . , X N , które są przedstawione za pomocą N wierzchołków grafu (skierowanego lub nieskierowanego, zależnie od zastosowania). Obecność krawędzi pomiędzy dwoma wierzchołkami świadczy w modelu o zależności warunkowej między powiązanymi z nimi zmiennymi. Z gaussowskim modelem graficznym mamy do czynienia, kiedy zmienne X 1 , X 2 , . . . , X N są z rozkładu normalnego. W tym przypadku pełna informacja o strukturze niezależności warunkowej między zmiennymi jest zawarta w macierzy odwrotnej do macierzy kowariancji (w macierzy precyzji) wektora losowego X = (X 1 , . . . , X N ) ′ . Czytelnika zainteresowanego głębszym zapoznaniem się z modelami graficznymi zachęcamy do książek wprowadzająch do tej teorii [START_REF] Lauritzen | Graphical models, volume 17 of Oxford Statistical Science Series[END_REF][START_REF] Whittaker | Graphical Models in Applied Multivariate Statistics[END_REF][START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. Dyskusję nad istnieniem MLE w dyskretnych modelach hierarchicznych, w tym w modelach graficznych, można znaleźć w pracy Wanga, Rauha i Massam [START_REF] Wang | Approximating faces of marginal polytopes in discrete hierarchical models[END_REF]. Odpowiednik estymatora LASSO w modelach graficznych został wprowadzony około piętnastu lat temu [START_REF] Meinshausen | High-dimensional graphs and variable selection with the Lasso[END_REF][START_REF] Yuan | Model selection and estimation in the Gaussian graphical model[END_REF][START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF] i jest w dalszym ciągu używany [START_REF] Brzyski | Selecting relevant groups of explanatory variables via convex optimization methods with the false discovery rate control[END_REF]. Graficzny odpowiednik SLOPE został wprowadzony niezależnie w pracy doktorskiej Sobczyka [START_REF] Sobczyk | Identifying low-dimensional structures through model selection in highdimensional data[END_REF] oraz w pracy Mazzy-Anthony'ego, Mazourego i Coatesa [START_REF] Mazza-Anthony | Learning Gaussian graphical models with ordered weighted ℓ 1 regularization[END_REF]. Wróćmy do struktury niezależności warunkowej w modelu graficznych. Można zauważyć, że położenie zer w macierzy precyzji wektora X jest identyczne z położeniem zer w laplasjanie grafu opisującego tę strukturę. Nasza dyskusja skupiona jest na tych z procesów Markowa, których dyskretyzacje mają macierz kowariancji równą odwrotności macierzy zbliżonej do laplasjanu zadanego grafu. Jako przykład można podać sytuację, w której zadany graf jest n-wierzchołkową ścieżką. Wtedy dodanie 1 do lewego górnego wyrazu laplasjanu skutkuje macierzą odwrotną do (Σ) i,j = min{i, j}, która to jest dyskretyzacją macierzy kowariancji

PL.5 Plan rozprawy

Rozprawa jest oparta na pięciu artykułach. Trzy z nich są opublikowane, jeden został wysłany do czasopisma, a kolejny jest w przygotowaniu i niebawem zostanie wysłany. Preprinty nieopublikowanych artykułów są dostępne w repozytoriach arXiv oraz HAL. [START_REF] Boucheron | Theory of classification: a survey of some recent advances[END_REF], dont la recherche a été lancée par Fisher [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF] et qui est appliquée dans presque tous les domaines liés aux données.

FR.1 Régression Linéaire Pénalisée

Parmi les applications importantes du problème de la classification, on peut citer les modifications de la régression linéaire, qui ne se concentrent pas sur la valeur exacte de l'estimation mais sur ses propriétés importantes.

= (X ′ X) -1 X ′ Y .
Si nous supposons en plus la gaussianité de Y , alors les OLS sont aussi l'estimateur du maximum de vraisemblance (MLE) de β [1, str. 28]. Lorsque la matrice de planification X est orthogonale, c'est-à-dire que X ′ X = cI p , c > 0, la formule ci-dessus se résume à βOLS = cX ′ Y . Sous les hypothèses ci-dessus, les OLS est également un estimateur fortement consistant de β [START_REF] Anderson | Strong consistency of least squares estimates in normal linear regression[END_REF]. Cependant, cet estimateur n'est pas défini de manière unique lorsque la matrice X ′ X est unité. défini lorsque la matrice [START_REF] Schwarz | Estimating the dimension of a model[END_REF] ou AIC [START_REF] Akaike | A new look at the statistical model identification[END_REF]. Une autre approche consiste à utiliser la régression pénalisée de la forme

β := arg min b ∈R p 1 2 ∥Y -Xb∥ 2 2 + pen λ (b) ,
où pen λ (b) est un pénalisateur arbitraire non négatif, modifiable en fonction de la valeur de λ > 0. Elle a été appliquée, par exemple, à l'imagerie cérébrale [START_REF] Brzyski | Connectivity-informed adaptive regularization for generalized outcomes[END_REF], à la prévision des prix de l'électricité [START_REF] Jedrzejewski | Electricity price forecasting: The dawn of machine learning[END_REF][START_REF] Uniejewski | Automated variable selection and shrinkage for day-ahead electricity price forecasting[END_REF] ou encore dans les mathématiques financières pour regrouper les actifs en fonction de leur corrélation partielle avec la série chronologique des rendements des fonds spéculatifs [START_REF] Kremer | Sparse index clones via the sorted ℓ 1 -norm[END_REF]. La première présence de cette idée est la sélection du meilleur sous-ensemble [START_REF] Beale | The discarding of variables in multivariate analysis[END_REF][START_REF] Hocking | Selection of the best subset in regression analysis[END_REF] avec pen λ (b) = λ∥b∥ 0 étant un nombre de coordonnées non nulles de b multiplié par λ. Cependant, pour de grandes valeurs de p, trouver le meilleur sous-ensemble par cette méthode est un problème NP-difficile [START_REF] Freijeiro-González | A critical review of LASSO and its derivatives for variable selection under dependence among covariates[END_REF]. Pour une solution plus parcimonieuse, on peut utiliser l'estimateus LASSO (Least Absolute Shrinkage and Selection Operator [START_REF] Chen | Basis pursuit[END_REF][START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF]), dans lequel la pénalité ajoutée à la somme des carrés des résidus

:= arg min b ∈R p 1 2 ∥Y -Xb∥ 2 2 + λ∥b∥ 1 .
L'estimateur LASSO n'est pas sans biais, en tant qu'estimateur de rétrécissement il apporte certains coefficients βLASSO j vers zéro. Certaines des coordonnées sont complètement réduites à zéro, ce qui donne une estimation plus parcimonie. Lorsque la matrice de planification X est orthonormée, c'est-à-dire X ′ X = I p , la formule exacte de βLASSO trouvée par Tibshirani [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] est basée sur βOLS :

FR.2. SLOPE xxv βLASSO i = sign( βOLS i ) max | βOLS i | -λ, 0 .
Pour garantir l'existence d'une solution du problème de la régression linéaire pénalisée, il y a nombreux propositions que la pénalité soit convexe. Cela permet également d'appliquer les outils de l'analyse convexe. Pour une comparaison plus large d'autres modifications d'un estimateur LASSO, nous renvoyons à l'article [START_REF] Freijeiro-González | A critical review of LASSO and its derivatives for variable selection under dependence among covariates[END_REF]. Cependant, il existe également des applications de pénalisateurs non convexes, par exemple la sélection du meilleur sous-ensemble [START_REF] Beale | The discarding of variables in multivariate analysis[END_REF][START_REF] Hocking | Selection of the best subset in regression analysis[END_REF] ou l'estimateur SCAD (Smoothly Clipped Absolute Deviation [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF]).

FR.2 SLOPE

Une autre approche pour réduire la dimensionnalité est l'estimateur pénalisé trié ℓ 1 (SLOPE [START_REF] Bogdan | Statistical estimation and testing via the sorted l1 norm[END_REF][START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF][START_REF] Zeng | Decreasing weighted sorted ℓ 1 regularization[END_REF]), qui, en plus de généraliser la méthode LASSO, regroupe les coefficients égaux de β et les colonnes corrélées de la matrice de planification X. Comme son nom l'indique, dans SLOPE, la norme ℓ 1 comme pénalisateur est remplacée par la norme ℓ 1 triée :

J Λ (b) := p i=1 |b| (i) λ i , où λ 1 > 0, λ 1 ≥ . . . ≥ λ p ≥ 0 et |b| (1) ≥ . . . ≥ |b| (p)
sont les valeurs absolues des coordonnées de b triées par ordre décroissant. Le sous-cas particulier de SLOPE avec Λ étant une séquence arithmétique est également connu sous le nom OSCAR [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF]. Il est important de noter que dans SLOPE, un seul paramètre de régularisation λ est remplacé par un vecteur non croissant Λ = (λ 1 , λ 2 , . . . , λ p ) ′ de p paramètres de régularisation (le vecteur de régularisation). Ainsi, il nous permet de modifier non seulement l'échelle de Λ, mais aussi sa forme. La principale motivation des auteurs de SLOPE était de tester les p hypothèses nulles H i 0 : β i = 0 et le contrôle du taux de fausses découvertes (le contrôle FDR), qui est défini par la proportion attendue entre le nombre de faux rejets d'hypothèses nulles et le nombre total de rejets d'hypothèses nulles (en cas d'absence de rejet, le FDR est défini comme égal à zéro). De plus, SLOPE généralise certaines des approches précédentes en régression linéaire : Le SLOPE schéma ne préserve pas seulement le signe d'un vecteur, mais détecte également ses clusters, c'est-à-dire les ensembles de coordonnées partageant la même valeur absolue et l'ordre entre ces valeurs absolues (ordre entre les clusters).

• λ 1 = . . . = λ p = 0 ⇒ βSLOPE = βOLS , • λ 1 = . . . = λ p > 0 ⇒ βSLOPE =
Fait FR.2.1 (Propriétés de base du SLOPE schéma [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]).

(a) pour chaque 1

≤ l ≤ ∥patt(b)∥ ∞ il existe j tel que |patt(b) j | = l, (b) sign(patt(b)) = sign(b) (préservation du signe), (c) |b i | = |b j | ⇒ |patt(b) i | = |patt(b) j | (préservation du cluster), (d) |b i | > |b j | ⇒ |patt(b) i | > |patt(b) j | (préservation de l'ordre).
Exemple FR.2.2. patt((4, 0, -1.5, 1.5, -4)

′ ) = (2, 0, -1, 1, -2) ′ .
On dit que l'estimateur SLOPE β SLOP E récupère le schéma de β lorsque

patt β SLOP E = patt(β).
Dans les chapitres 3 et 4, nous discutons de nouvelles conditions nécessaires et suffisantes pour le recouvrement du SLOPE schéma, ainsi que de nouveaux résultats pour n ≥ p sur la consistance et la forte consistance de l'estimation SLOPE et de son schéma.

FR.2.1 Consistance

L'outil principal pour montrer la consistance forte de SLOPE est la forte consistance de l'estimateur des moindres carrés, dont la preuve a été présentée par exemple dans l'article de Anderson et Taylor [START_REF] Anderson | Strong consistency of least squares estimates in normal linear regression[END_REF]. Les principaux résultats sur la consistance de LASSO peuvent être trouvés dans les articles suivants:

• consistance de LASSO: Knight, Fu, 2000 [START_REF] Knight | Asymptotics of Lasso-type estimators[END_REF],

• forte consistance de LASSO : Chatterjee, Lahiri, 2011 [START_REF] Chatterjee | Strong consistency of Lasso estimators[END_REF], Les principales hypothèses sont que le paramètre de régularisation λ n est d'un ordre inférieur à n et qu'il existe une espérance infinie de la valeur absolue d'un terme d'erreur.

• consistance du signe de LASSO: Zhao, Yu, 2006 [START_REF] Zhao | On model selection consistency of Lasso[END_REF].

Les sections 3.

(-1,2) (1,2) (2,1) (2,-1) (1,-2) (-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1) (1,0) (1,-1) (0,-1) (-1,-1) (0,0) βSLOPE et βOLS dans un plan orthogonal: X ′ X = I p pour Λ = (2, 1) ′ .

FR.2.2 Condition d'irreprésentabilité

Étant donné tout n, p > 0, dans LASSO, la consistance de la sélection du vrai sous-ensemble de β est presque équivalente à ce que les coordonnées en dehors du vrai support de β ne soient pas représentables par les coordonnées à l'intérieur du support [START_REF] Zhao | On model selection consistency of Lasso[END_REF]. Nous introduisons un analogue de cette condition d'irreprésentabilité de LASSO à SLOPE. Puis nous dérivons la caractérisation géométrique d'une estimation SLOPE.

FR.2.3 Géométrie de la régression linéaire pénalisée

Pour une meilleure compréhension des connexions entre βSLOPE et βOLS , nous donnons une illustration d'un cas de bas dimension p ≤ n pour p = 2. Ici, l'estimation SLOPE est une différence entre l'estimation des moindres carrés et sa projection sur (X ′ X) -1 C Λ , où C Λ est une boule unité dans une norme double de J Λ . Ce résultat est particulièrement facile à interpréter dans le cas où X est une matrice orthogonale. Il en résulte une formule plus simple, qui a été récemment proposée par Tardivel, Servien et Concordet [START_REF] Tardivel | Simple expressions of the lasso and slope estimators in low-dimension[END_REF]. Pour une présentation plus large de l'application de l'approche géométrique à l'estimateur SLOPE et à sa récouvrement de schémas, nous invitons à la Section 4.4. LASSO et SLOPE présentent une connexion avec la théorie des systèmes racines. En particulier, on peut observer qu'une boule unité en norme ℓ ∞ , qui est duale de la norme ℓ 1 utilisée dans LASSO et une boule unité en norme J * Λ sont proportionnelles, respectivement, aux coques convexes des orbites des groupes de Weyl correspondant au produit cartésien p ieme du système racine A 1 et au système racine B p [START_REF] Godland | Projections and angle sums of permutohedra[END_REF][START_REF] Postnikov | Permutohedra, associahedra, and beyond[END_REF]. Pour plus d'informations sur la théorie des systèmes racine et ses recherches actuelles, nous renvoyons à un cahier de cours de Helgason [START_REF] Helgason | Differential geometry, Lie groups, and symmetric spaces[END_REF] et à [START_REF] Graczyk | Potential kernels for radial Dunkl Laplacians[END_REF][START_REF] Dziubański | Remark on atomic decompositions for the Hardy space H 1 in the rational Dunkl setting[END_REF]. La géométrie des estimateurs de régression linéaire pénalisés et leur recouvrement peuvent être inspectés dans un cas plus général, ce qui constitue l'essence du chapitre 5. En effet, l'estimateur SLOPE peut être classé dans l'une des classes d'un nombre fini de classes lorsque la pénalité est une gauge polyédrique, c'est-à-dire une fonction convexe non négative, qui est le maximum d'un nombre fini de fonctions linéaires. Les caractéristiques importantes de cette estimation, aussi appelée son schéma, peuvent être entièrement décrites avec le sous-

β 1 β 2 OLS SLOPE (1,2) (2,1) (-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1) (1,0) (1,-1) (0,-1) (-1,-1) (0,0) βSLOPE et βOLS for X ′ X = 1 0.6 0.6 1 et Λ = (2, 1) ′ .
différentiel du pénaliseur. Puisque les inégalités pour les fonctions linéaires déterminant le gauge polyédrique sont satisfaites par une intersection du nombre fini de demi-espaces correspondants, ces estimateurs sont fortement liés à la théorie des polytopes, cf. les livres de Gruber [START_REF] Gruber | Convex and Discrete Geometry[END_REF], Grünbaum [START_REF] Grünbaum | Convex polytopes[END_REF], Hiriart-Urruty et Lemarechal [START_REF] Hiriart-Urruty | Convex Analysis and Minimization Algorithms I: Fundamentals[END_REF] ou Ziegler [START_REF] Ziegler | Lectures on Polytopes[END_REF]. Dans le chapitre 5, nous généralisons certains des nouveaux résultats pour SLOPE à la classe d'estimateurs ci-dessus. Pour généraliser la notion du schéma à la classe des gauges polyédriques, nous considérons la classe d'équivalence des patrons, ce qui nous permet de désigner son recouvrement, comme le recouvrement de la sous-différentielle de pen. Nous introduisons et caractérisons l'accessibilité du schéma et nous donnons une condition d'irreprésentabilité pour les gabarits polyédriques. Avec des outils issus de la géométrie polyédrique, en particulier avec les cônes normaux des faces des polyèdres considérés, nous discutons les propriétés de l'égalité des schémas entre vecteurs. Nous étudions également les relations entre les ensembles de vecteurs ayant le même schéma avec la notion de sous-espace modèle, cf. [START_REF] Vaiter | Model selection with low complexity priors[END_REF]. Plus tard, nous discutons de la récouvrement du schéma par les estimateurs pénalisés seuillés, qui sont une généralisation du LASSO seuillé, qui ne considère pas seulement le signe de l'estimateur, mais son schéma entier. Notre discussion étend également la caractérisation de l'unicité uniforme d'un estimateur de la classe des normes polyédriques [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] aux gauges polyédriques.

FR.3 Familles exponentielles discrètes

Dans le chapitre 6, nous passons des espaces euclidiens aux espaces discrets, c'est-à-dire les espaces avec un nombre fini d'éléments. Ici, le point majeur de notre intérêt sont les familles exponentielles discrètes, que nous comprenons comme des familles exponentielles définies sur un espace fini d états X , |X | = K < ∞. Pour les familles sur les ensembles infinis dénombrables, voir Jacobsen [START_REF] Jacobsen | Existence and unicity of MLEs in discrete exponential family distributions[END_REF]. Nous considérons ensuite un sous-espace linéaire B de l'espace des fonctions linéaires R X , qui contient une fonction constante strictement positive. Nous introduisons également une fonction de poids strictement positive µ : X → (0, ∞). Nous construisons la famille exponentielle de la façon suivante: De plus, pour les applications proposées, nous utilisons les limites de probabilité établies par les outils classiques de la théorie des graphes aléatoires et du problème du collecteur de coupons, voir [118, p. 194-195], [START_REF] Erdős | On a classical problem of probability theory[END_REF], et [START_REF] Pósfai | Approximation theorems related to the coupon collector's problem[END_REF]. Dans la famille des fonctions de Walsh, nous utilisons les propriétés de l'hypercube {-1, 1} k (k = log 2 K), de ses sous-cubes et du graphe hypercube correspondant.

FR.4 Modèles graphiques

L'une des branches de la statistique qui est connue à la fois pour l'utilisation de l'MLE et pour son efficacité et pour appliquer l'estimation pénalisée est la théorie des modèles graphiques. Un modèle graphique est une famille de distributions de probabilités d'une collection finie de variables aléatoires X 1 , . . . , X N , qui sont codées par N sommets d'un graphe (dirigé ou non dirigé). La présence (ou l'absence) d'une arête entre deux sommets renseigne sur la dépendance (ou l'indépendance) conditionnelle entre les sommets correspondants. Lorsque toutes les variables aléatoires sont gaussiennes, on parle de modèles graphiques gaussiens. Dans ce cas, la structure d'indépendance conditionnelle est entièrement codée par l'emplacement des zéros dans l'inverse de la matrice de covariance (la matrice de précision) d'un vecteur X = (X 1 , . . . , X N ) ′ . Par savoir plus sur les modèles graphiques nous invitons à consulter les introductions existantes sur ce sujet [START_REF] Lauritzen | Graphical models, volume 17 of Oxford Statistical Science Series[END_REF][START_REF] Whittaker | Graphical Models in Applied Multivariate Statistics[END_REF][START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. Pour la discussion sur l'existence de MLE dans les modèles hiérarchiques discrets, qui incluent les modèles graphiques, nous nous référons à l'article de Wang, Rauh et Massam [START_REF] Wang | Approximating faces of marginal polytopes in discrete hierarchical models[END_REF]. La version modèle graphique d'un estimateur LASSO a été proposée il y a environ quinze ans [START_REF] Meinshausen | High-dimensional graphs and variable selection with the Lasso[END_REF][START_REF] Yuan | Model selection and estimation in the Gaussian graphical model[END_REF][START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF] et gagne encore en popularité, cf. par exemple [START_REF] Brzyski | Selecting relevant groups of explanatory variables via convex optimization methods with the false discovery rate control[END_REF]. La SLOPE graphique a été proposée indépendamment par Sobczyk dans sa thèse de doctorat [START_REF] Sobczyk | Identifying low-dimensional structures through model selection in highdimensional data[END_REF] et par Mazza-Anthony, Mazoure et Coates [START_REF] Mazza-Anthony | Learning Gaussian graphical models with ordered weighted ℓ 1 regularization[END_REF]. Revenons-en à la structure d'indépendance conditionnelle. On peut observer que l'emplacement des zéros dans la matrice de précision de X est le même que dans la matrice laplacienne d'un graphe sous-jacent. Dans notre discussion, nous examinons de plus près de tels graphes et recherchons ces processus de Markov, dont les discrétisations ont une matrice de covariance qui est l'inverse d'une matrice laplacienne un peu modifiée. À titre d'exemple, lorsqu'un graphique sous-jacent est un chemin de n sommets, l'ajout de 1 à l'entrée supérieure gauche de sa matrice laplacienne donne une inverse égale à (Σ) i,j = min{i, j}, qui est une version discrétisée de la matrice de covariance d'un processus de Wiener. Chapter 1

FR.5 Plan de la thèse

Introduction

In recent times we may observe the rapid and inevitable emergence of data collection, both in the number of observations and in the number of features to measure them. Due to the increase of the influence of big data on everyday life, mathematical statistics and data science gain growth of interest in mathematics and applied sciences. However, they should not be considered separately from other branches of mathematics. In this dissertation we focus on proposing new connections of modern statistics with geometry and combinatorics. We start our discussion with finite-dimensional Euclidean spaces. We pay attention to those situations where the space of possible values of the estimator can be partitioned into a finite number of subsets such that there is a one-to-one correspondence between the subsets and specific properties of estimates. This prediction of the properties of an observation is better known as a classification problem [START_REF] Boucheron | Theory of classification: a survey of some recent advances[END_REF], research on which was started by Fisher [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF] and is being applied in almost all fields related to data processing.

Penalized Linear Regression

As one of important applications of the classification problem, we can point out the modifications of linear regression, which do not focus on the exact estimate value but on its important properties. In the multiple linear regression model having n observations and p explanatory variables, we assume that the response vector

Y = (Y 1 , . . . , Y n ) ′ ∈ R n is of the form Y = Xβ + ε, where X ∈ R n×p is a design matrix, β = (β 1 , . . . , β p ) ′ ∈ R p is an unknown vector of regression coefficients and ε = (ε 1 , . . . , ε n ) ′ ∈ R n is a random noise.
The main objective of linear regression is to estimate β and recover its essential properties. The wide class of problems of choosing the subset of coordinates of β, which is most suitable for our demands on the estimate and its sparsity, is better known as the model selection problem. For the survey on the model selection problem, cf. [START_REF] Claeskens | Statistical model choice[END_REF]. In most of our discussion, we assume that the error ε has a symmetric and continuous distribution. Some of our results cover the noiseless case of ε ≡ 0, which is a strong tool for establishing asymptotic results. Due to the abundance of the existing knowledge on the Gaussian distribution, we are also able to present more efficient results on the behavior of considered estimators with Gaussian ε ∼ N (0, σ 2 I n ). The most classical linear regression method, the Ordinary Least Squares method (OLS), was proposed by Legendre at the beginning of the nineteenth century [START_REF] Barbin | Histoires de probabilités et de statistiques[END_REF][START_REF] Hald | A history of parametric statistical inference from Bernoulli to Fisher, 1713-1935[END_REF][START_REF] Legendre | Nouvelles méthodes pour la détermination des orbites des comètes. Nineteenth Century Collections Online[END_REF]. The OLS estimator is defined as the minimizer of the residual sum of squares, namely βOLS := arg min

b ∈R p ∥Y -Xb∥ 2 .
The OLS estimator gained its popularity in the case n ≥ p due to its simplicity, efficiency for a relatively small number of variables [33, p. 4] and many useful statistical properties. For example, if the matrix X ′ X is invertible and Cov(Y ) = σ 2 I n for σ > 0, then the OLS is the best linear unbiased estimator (BLUE) [1, Theorem 2.7.1.] of β. In this case, the exact formula for βOLS can be easily deduced [1, p. 28]:

βOLS = (X ′ X) -1 X ′ Y .
If we additionally assume the gaussianity of Y , then OLS is also the maximum likelihood estimator (MLE) of β [1, p. 28]. When the design matrix X is orthogonal, i.e. X ′ X = cI p , c > 0, the above formula boils down to βOLS = 1 c X ′ Y . Under the above assumptions, the OLS is also a strongly consistent estimator of β [START_REF] Anderson | Strong consistency of least squares estimates in normal linear regression[END_REF]. However, this estimator is not uniquely defined when the matrix X ′ X is not invertible, which is the case in a high-dimensional setting p > n.

Additionally, under common assumptions on the error term ε, OLS estimate is not sparse, with probability 1 it contains p coordinates of mutually different values. For an example of unusual assumptions on Y and X and no unique estimate, one may get acquainted with to a recent article by Dupuis and Vaiter [START_REF] Dupuis | The geometry of sparse analysis regularization[END_REF]. As the true vector β ∈ R p of regression coefficients could contain much fewer nonzero coordinates, there is a natural pathway to propose methods that would promote the sparsity of β, that is, a small number of nonzero coordinates, or at least to promote the estimates of β that are describable by a small number of parameters. Several solutions were proposed to deal with such problem. One of them is to compare the suitable models by an information criterion, for example BIC [START_REF] Schwarz | Estimating the dimension of a model[END_REF] or AIC [START_REF] Akaike | A new look at the statistical model identification[END_REF]. Another approach is to use the penalized regression of the form

β := arg min b ∈R p 1 2 ∥Y -Xb∥ 2 2 + pen λ (b) ,
where pen λ (b) is an arbitrary nonnegative penalizer, modifiable according to the value of λ > 0.

It has been applied, for example, in brain imaging [START_REF] Brzyski | Connectivity-informed adaptive regularization for generalized outcomes[END_REF], forecasting electricity prices [START_REF] Jedrzejewski | Electricity price forecasting: The dawn of machine learning[END_REF][START_REF] Uniejewski | Automated variable selection and shrinkage for day-ahead electricity price forecasting[END_REF] or in financial mathematics to group assets with respect to their partial correlation with the hedge fund return times series [START_REF] Kremer | Sparse index clones via the sorted ℓ 1 -norm[END_REF]. The first presence of this idea is the best subset selection [START_REF] Beale | The discarding of variables in multivariate analysis[END_REF][START_REF] Hocking | Selection of the best subset in regression analysis[END_REF] with pen λ (b) = λ∥b∥ 0 being a number of nonzero coordinates of b multiplied by λ. However, for large values of p, finding the best subset by this method is an NP-hard problem [START_REF] Freijeiro-González | A critical review of LASSO and its derivatives for variable selection under dependence among covariates[END_REF]. For a sparser solution, one may use the Least Absolute Shrinkage and Selection Operator (LASSO [START_REF] Chen | Basis pursuit[END_REF][START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF]), in which the penalty added to the residual sum of squares ∥Y -X β∥ 2 2 is an ℓ 1 norm of β and the tuning parameter is λ > 0:

βLASSO := arg min b ∈R p 1 2 ∥Y -Xb∥ 2 2 + λ∥b∥ 1 .
The LASSO estimator is not unbiased, as a shrinkage estimator it brings some coefficients βLASSO j toward zero. Some of the coordinates are being shrunk completely to zero, which results in a sparser estimate. When the design matrix X is orthonormal, i.e. X ′ X = I p , the exact formula for βLASSO found by Tibshirani [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] is based on βOLS :

βLASSO i = sign( βOLS i ) max | βOLS i | -λ, 0 .
To guarantee the existence of a solution of the penalized linear regression problem, many proposals offer the penalty to be convex. This also allows one to apply the tools from convex analysis. For a broader comparison of other modifications of a LASSO estimator, we refer to the article [START_REF] Freijeiro-González | A critical review of LASSO and its derivatives for variable selection under dependence among covariates[END_REF]. However, there are also applications of nonconvex penalizers, e.g., the best subset selection [START_REF] Beale | The discarding of variables in multivariate analysis[END_REF][START_REF] Hocking | Selection of the best subset in regression analysis[END_REF] or the smoothly clipped absolute deviation (SCAD) [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF].

SLOPE

Another approach to reduce dimensionality is the Sorted ℓ 1 Penalized Estimator (SLOPE [START_REF] Bogdan | Statistical estimation and testing via the sorted l1 norm[END_REF][START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF][START_REF] Zeng | Decreasing weighted sorted ℓ 1 regularization[END_REF]), which apart of generalizing the LASSO method, clusterizes the equal coefficients of β and correlated columns of the design matrix X. As the name suggests, in SLOPE the ℓ 1 norm as penalizer is replaced by the sorted ℓ 1 norm:

J Λ (b) := p i=1 |b| (i) λ i ,
where λ 1 > 0, λ 1 ≥ . . . , λ p ≥ 0 and |b| (1) ≥ . . . ≥ |b| (p) are the absolute values of coordinates of b sorted in descending order. The special subcase of SLOPE with Λ being an arithmetic sequence is also known as the OSCAR estimator [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF]. It is important to note that in SLOPE a single tuning parameter λ is replaced by a non-increasing vector Λ = (λ 1 , λ 2 , . . . , λ p ) ′ of p tuning parameters (the tuning vector). Thus, it allows us to modify not only the scaling of Λ, but also its shape. The main motivation of the authors of SLOPE was the testing of p null hypotheses H 0 i : β i = 0 and the control of the rate of false discoveries (the FDR control), which is defined as the expected proportion between the amount of false rejections of null hypotheses and the total amount of rejections of null hypotheses (in case of no rejections, the FDR is defined to equal zero). Moreover, SLOPE generalizes some of the previous approaches in linear regression:

• λ 1 = . . . = λ p = 0 ⇒ βSLOPE = βOLS , • λ 1 = . . . = λ p > 0 ⇒ βSLOPE = βLASSO ,
• Λ is an arithmetic sequence ⇒ βSLOPE = βOSCAR .

A more detailed description of the history of research on SLOPE can be found in Section 4.1.1. In our research, we focus on another important property of SLOPE, that is the recovery of important characteristics of a vector of regression coefficients β, called its SLOPE pattern. It is an analogue of the sign vector in LASSO and, like its precedessor, it is fully describable by the subdifferential of the penalizing norm. To be more specific, let k be the number of clusters of patt(β) = (m 1 , . . . , m p ) ′ i.e., the number of distinct nonzero components of |β|. The SLOPE pattern does not only preserve the sign of a vector, but also detects its clusters, i.e. the sets of coordinates sharing the same absolute value and the order between those absolute values (hierarchy between clusters).

|b i | = |b j | ⇒ |patt(b) i | = |patt(b) j | (cluster preservation), (d) |b i | > |b j | ⇒ |patt(b) i | > |patt(b) j | (hierarchy preservation).
Example 1.2.2. patt((4, 0, -1.5, 1.5, -4) ′ ) = (2, 0, -1, 1, -2) ′ .

We say that the SLOPE estimator β SLOP E recovers the pattern of β when patt β SLOPE = patt(β).

In Chapter 3 and Chapter 4 we discuss novel necessary and sufficient conditions for the recovery of the SLOPE pattern, as well as the novel results for n ≥ p on the consistency and strong consistency of both the SLOPE estimate and its SLOPE pattern.

Consistency

The main tool to show the strong consistency of SLOPE is the strong consistency of the Ordinary Least Squares estimator, the proof of which was presented e.g. in the article of Anderson and Taylor [START_REF] Anderson | Strong consistency of least squares estimates in normal linear regression[END_REF]. Main results on the consistency of LASSO might be found in the following articles:

• consistency of LASSO: Knight, Fu, 2000 [START_REF] Knight | Asymptotics of Lasso-type estimators[END_REF],

• strong consistency of LASSO: Chatterjee, Lahiri, 2011 [START_REF] Chatterjee | Strong consistency of Lasso estimators[END_REF], Main assumptions are that the tuning parameter λ n is of a smaller order than n and that there is a finite expectation of the absolute value of an error term.

• consistency of the sign of LASSO: Zhao, Yu, 2006 [START_REF] Zhao | On model selection consistency of Lasso[END_REF].

Sections 3.4 and 4.6 focus on the strong consistency of SLOPE and its pattern. It is important to recall that if the tuning vector Λ is constant (LASSO as the special case of SLOPE), then the SLOPE pattern is not consistent, even if the assumption on the consistency of the sign vector is satisfied. As an example, the clusters of a true vector of parameters, which are contained in a SLOPE pattern, are not preserved by LASSO. As another interesting remark, it may be mentioned that the strong consistency of LASSO (or SLOPE) does not necessarily imply the strong consistency of its sign (of its SLOPE pattern). As a counterexample, one may notice that the sequence ((1/n, 1/n, -1/n, -1/n) ′ ) n≥1 converges to (0, 0, 0, 0) ′ , while its sign and its SLOPE pattern are equal (1, 1, -1, -1) ′ for every positive n.

Irrepresentability condition

Given any n, p > 0, in LASSO, the consistency of selection of the true subset of β is almost equivalent to the coordinates outside the true support of β not being representable by the coordinates inside the support [START_REF] Zhao | On model selection consistency of Lasso[END_REF]. We introduce an analogue of this Irrepresentability Condition from LASSO to SLOPE. Then we derive the geometric characterization of a SLOPE estimate.

Geometry of penalized linear regression

For a better understanding of the connections between βSLOPE and βOLS , below we give an illustration of a low-dimensional case p ≤ n for p = 2. Here, the SLOPE estimate is a difference between the least squares estimate and its projection onto (X ′ X) -1 C Λ , where C Λ is a unit ball in a norm dual to J Λ . This result is especially easy to interpret in the case of X being an orthogonal matrix. That resulted in an easier formula, which was recently proposed by Tardivel,

β 1 β 2 SLOPE OLS SLOPE OLS SLOPE OLS (-2,1) (-1,2) (1,2) (2,1) (2,-1) (1,-2) (-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1)
(1,0)

(1,-1) (0,-1) (-1,-1) (0,0)

β 1 β 2 OLS SLOPE (1,2) (2,1) (-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1)
(1,0)

(1,-1) (0,-1) (-1,-1) (0,0) Servien and Concordet [START_REF] Tardivel | Simple expressions of the lasso and slope estimators in low-dimension[END_REF]. For a wider presentation of applying the geometrical approach to the SLOPE estimator and its pattern recovery we invite the reader to Section 4.4. LASSO and SLOPE estimators exhibit a connection to the theory of root systems. Namely, it may be observed that a unit ball in ℓ ∞ norm, which is dual to the ℓ 1 norm used in LASSO and a unit ball in J * Λ norm are proportional, respectively, to the convex hulls of orbits of Weyl groups corresponding to the p th Cartesian product of the root system A 1 and of the root system B p [START_REF] Godland | Projections and angle sums of permutohedra[END_REF][START_REF] Postnikov | Permutohedra, associahedra, and beyond[END_REF]. For more on the theory of root systems and its current research, we refer to a coursebook of Helgason [START_REF] Helgason | Differential geometry, Lie groups, and symmetric spaces[END_REF] and to [START_REF] Graczyk | Potential kernels for radial Dunkl Laplacians[END_REF][START_REF] Dziubański | Remark on atomic decompositions for the Hardy space H 1 in the rational Dunkl setting[END_REF]. The geometry of penalized linear regression estimators and their pattern recovery may be inspected in more general case, which is the essence of Chapter 5. Indeed, the SLOPE estimator can be classified into one of finite number of classes when the penalty is a polyhedral gauge, i.e. a nonnegative convex function, which is a maximum of a finite number of linear functions. The important features of this estimate, also named its pattern, may be fully described with the subdifferential of the penalizer. Since the inequalities for linear functions determining the polyhedral gauge are satisfied by an intersection of the finite number of corresponding half-spaces, those estimators are strongly connected with the theory of polytopes, cf. the books of Gruber [START_REF] Gruber | Convex and Discrete Geometry[END_REF], Grünbaum [START_REF] Grünbaum | Convex polytopes[END_REF], Hiriart-Urruty and Lemarechal [START_REF] Hiriart-Urruty | Convex Analysis and Minimization Algorithms I: Fundamentals[END_REF] or Ziegler [START_REF] Ziegler | Lectures on Polytopes[END_REF].

In Chapter 5 we generalize some of the new results for SLOPE to the above class of estimators. To generalize the notion of pattern to the class of polyhedral gauges, we consider the pattern equivalence class, which allows us to denote its recovery, as the recovery of the subdifferential of pen. We introduce and characterize the accessibility of the pattern and we give an irrepresentability condition for polyhedral gauges. With tools from polyhedral geometry, especially with the normal cones of faces of considered polyhedra, we discuss the properties of the pattern equality between vectors. We also investigate the relations between sets of vectors having the same pattern with the notion of the model subspace, cf. [START_REF] Vaiter | Model selection with low complexity priors[END_REF]. Later on, we discuss the pattern recovery by thresholded penalized estimators, which are a generalization of the thresholded LASSO, which considers not only the sign of the estimator, but its entire pattern, Our discussion also extends the characterization of the uniform uniqueness of an estimator from the class of polyhedral norms [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] to polyhedral gauges.

Discrete exponential families

In Chapter 6 we move our focus from Euclidean to discrete spaces, i.e. spaces with a finite number of elements. Here, the major point of our interest are the discrete exponential families, which we understand as exponential families defined on a finite space of states X , |X | = K < ∞. For families on infinite countable sets, see Jacobsen [START_REF] Jacobsen | Existence and unicity of MLEs in discrete exponential family distributions[END_REF]. Then we consider a linear subspace B of the space of linear functions R X , which contains a strictly positive constant function. We also introduce a strictly positive weight function µ : X → (0, ∞).We construct the exponential family in a following way: For a real-valued function ϕ we define the partition and log-partition functions,

Z(ϕ) = x∈X e ϕ(x) µ(x), ψ(ϕ) = log Z(ϕ),
respectively, and exponential density p = e(ϕ) = e ϕ-ψ(ϕ) = e ϕ /Z(ϕ).

GRAPHICAL MODELS

The exponential family spanned by B is e(B) := {p = e(ϕ) : ϕ ∈ B}. This allows us to derive the likelihood and log-likelihood function. The log-likelihood function is strictly concave. Therefore, if the Maximum Likelihood Estimator (MLE) exists, then it is unique. Despite the boundedness of the likelihood function, MLE may not exist. Our main goal is to establish a new characterization of the existence of MLE and to apply it to specific families, exponential families spanned by Rademacher and Walsh functions, and exponential families of random graphs.

The key tool used in our discussion is the newly introduced notion of set of uniqueness, that is such set U ⊂ X , that ϕ = 0 is the only function of a given class of functions that equal zero on U . To be more specific, we show that the MLE for e(B) and the i.i.d. sample x 1 , x 2 , . . . , x n ∈ X exists if and only if {x 1 , . . . , x n } is the set of uniqueness for the nonnegative cone B + := {ϕ ∈ B : ϕ ≥ 0}. We also restate this criterion as a linear programming problem. Additionally, for proposed applications, we use the probability bounds established by classical tools from random graph theory and from the Coupon Collector's Problem, see [118, pp. 194-195], [START_REF] Erdős | On a classical problem of probability theory[END_REF], and [START_REF] Pósfai | Approximation theorems related to the coupon collector's problem[END_REF]. In the family spanned by Walsh functions, we use the properties of the hypercube {-1, 1} k (k = log 2 K), of its subcubes and of the corresponding hypercube graph.

Graphical models

One of those branches of statistics that is known both for using the MLE and for applying penalized estimation is the theory of graphical models. A graphical model is a family of probability distributions of a finite collection of random variables X 1 , X 2 , . . . , X N , which are encoded by N vertices of a graph (directed or undirected). The presence (or absence) of an edge between two vertices informs about the conditional dependence (or independence) between the corresponding vertices. When all random variables are Gaussian, we refer to Gaussian graphical models. In this case, the conditional independence structure is fully encoded by the location of zeros in the inverse of the covariance matrix (the precision matrix) of a vector X = (X 1 , . . . , X N ) ′ . We invite the reader interested in graphical models to the existing introductions for this topic [START_REF] Lauritzen | Graphical models, volume 17 of Oxford Statistical Science Series[END_REF][START_REF] Whittaker | Graphical Models in Applied Multivariate Statistics[END_REF][START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. For the discussion on the existence of MLE in discrete hierarchical models, which include graphical models, we refer to the article of Wang, Rauh and Massam [START_REF] Wang | Approximating faces of marginal polytopes in discrete hierarchical models[END_REF]. The graphical model version of a LASSO estimator was proposed around fifteen years ago [START_REF] Meinshausen | High-dimensional graphs and variable selection with the Lasso[END_REF][START_REF] Yuan | Model selection and estimation in the Gaussian graphical model[END_REF][START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF] and still gains popularity, cf. e.g. [START_REF] Brzyski | Selecting relevant groups of explanatory variables via convex optimization methods with the false discovery rate control[END_REF]. The graphical SLOPE was proposed independently by Sobczyk in his Ph.D. dissertation [START_REF] Sobczyk | Identifying low-dimensional structures through model selection in highdimensional data[END_REF] and by Mazza-Anthony, Mazoure and Coates [START_REF] Mazza-Anthony | Learning Gaussian graphical models with ordered weighted ℓ 1 regularization[END_REF]. Let us come back to the conditional independence structure. It may be observed that the location of zeros in the precision matrix of X is the same as in the Laplacian matrix of an underlying graph. In our discussion we take a closer look at such graphs and look for those Markov processes, the discretizations of which have a covariance matrix being an inverse of a slightly modified Laplacian matrix. As an example, when an underlying graph is a path of n vertices, then adding 1 to the top-left entry to its Laplacian matrix results in an inverse equal to (Σ) i,j = min{i, j}, which is a discretized version of a covariance matrix of a Wiener process.

Plan of the dissertation

The dissertation is based on five articles, three of which are published, one is currently under the review process, and one is an extended version of a preprint, which can be found on HAL and will be submitted soon. The outline of the dissertation is the following: Chapter 2 contains preliminaries of the presented research and describes the basic and novel notions used in latter chapters.

In Chapter 3 we focus on the SLOPE estimation in case of design matrix being orthogonal, that is, X ′ X = nI p . We also present new results on the strong consistency of the SLOPE estimators and on the strong consistency of pattern recovery by SLOPE when the design matrix is orthogonal and illustrate the advantages of the SLOPE clustering in the context of high frequency signal denoising. Chapter 4 focuses on the SLOPE pattern recovery with no restrictions on the design matrix.

Here, we show that recovery of the pattern can be determined by two criteria, called the positivity condition and the subdifferential condition. Later, we introduce a SLOPE irrepresentability condition, which generalizes the well known LASSO irrepresentability condition [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF][START_REF] Bühlmann | Statistics for High-Dimensional Data: Methods, Theory and Applications[END_REF] and then we illustrate it geometrically. Later on, we derive the refined asymptotic conditions on both consistency and strong consistency of the SLOPE estimate and of its pattern. Chapter 5 concerns the problem of pattern recovery in a general case of a penalty being a polyhedral gauge. We establish the necessary and sufficient condition for the uniform uniqueness of the estimator. Then we derive conditions for the pattern recovery of polyhedral gauge penalized estimators and for their thresholded versions. Finally, we generalize the irrepresentability condition for penalizers being polyhedral gauges and discuss the connections between a pattern and a model from the article by Vaiter et al. [START_REF] Vaiter | Model selection with low complexity priors[END_REF]. Chapter 6 focuses on the necessary and sufficient conditions for the existence of the Maximum Likelihood Estimator (MLE) in discrete exponential families. The main tool in this article is the notion of sets of uniqueness, i.e. such subsets of a state space on which a function from a given class equals zero if and only if it equals zero on a whole space. We apply our results to the class of Rademacher and Walsh functions and to exponential random graph models (ERGM). In Chapter 7 we discuss the connection between Gaussian graphical models and the covariance matrices of discretized Wiener processes. For that reason, we use the discrete Laplacian matrix of an underlying graphical structure.

Chapter 2

Preliminaries and basic notions on penalized linear regression

Notation

The content of this section will be completed after unifying the notation in the whole dissertation.

• A ′ -transpose of a matrix A

• |b| ↓ -nonincreasing permutation of absolute values of coordinates of b • ∂f (x) -subdifferential of f at x • B ∥•∥ (b, r) -ball of center b and radius r in norm ∥ • ∥ • cl(C) -closure of C • int(C) -interior of C • aff(C) -affine hull of (C) • - → C -unique linear space parallel to C • cl(C) -closure of C • bd(C) -boundary of C • ri(C) -relative interior of C • rb(C) -relative boundary of C • conv(C) -convex hull of C • cone(C) -conic hull of C • rec(C) -recession cone of C • P C (x) -orthogonal projection of x onto C • C * -polar set of C • 1 k = (1, . . . , 1) ′ ∈ R k 11 • 1 A (x) = 1, x ∈ A, 0, x / ∈ A.
-characteristic function of a set A For the convenience of the reader, when a set A is defined as a set of points satisfying the relation ϕ we reduce the notation from 1 {x|ϕ(x)} (x) to 1 {ϕ(x)} , e.g. 1 {x>0} .

•

J n ∈ R n×n , (J n ) ij = 1 for every 1 ≤ i, j ≤ n. • supp(v) = {i ∈ {1, . . . , p}| s i ̸ = 0} -support of a vector v ∈ R p • diag(S) ∈ R p×p -diagonal matrix with S ∈ R p on the diagonal (diag(S)) i,j = s i 1 {i=j} • M p -set of all possible SLOPE patterns of b ∈ R p • sign(x) :=        1 if x > 0 0 if x = 0 -1 if x < 0 -sign function • sign(x) = (sign(x 1 ), . . . , sign(x p )) ′ -sign vector • R k+ := {κ ∈ R k : κ 1 > κ 2 > . . . > κ k > 0}
• S p -symmetric permutation group on the set {1, . . . , p}

• Q -:= max{Q, 0}.

Penalized linear regression 2.2.1 Linear regression

We consider the following linear regression model

Y = Xβ + ε,
where Y ∈ R n is a response vector with n being a sample size, X ∈ R n×p is a design matrix with p being the number of unknown parameters, β ∈ R p is an unknown parameter vector and ε ∈ R n is a noise term.

The case of ε = 0 (ε ̸ = 0) is referred later as a noiseless (noisy) case.

In the penalized linear regression we want to find an estimator β of a vector β, which is of the form

β := arg min b ∈R p 1 2 ∥Y -Xb∥ 2 2 + pen Λ (b) ,
where, depending of the estimation method, Λ is a tuning parameter or a vector of tuning parameters (tuning vector). When dealing with a single tuning parameter, we denote it with a small letter λ, restricting a capital letter Λ for a vector (λ 1 , . . . , λ p ) ′ . As we are mostly interested with recovering a pattern of a vector β, we are interested to define it in a general case. For that reason, we use a notion of a subdifferential and define two vectors to have the same pattern if the penalizer has the same subdifferential at those vectors Definition 2.2.1 (Subdifferential). [41, p. 76] Let f : R p → R be a convex function. A vector

d ∈ R p is called a subgradient of f at point x ∈ R p if f (y) ≥ f (x) + (y -x) ′ d, ∀ y ∈ R p .
The set of all subgradients of the convex function f at x ∈ R p is called the subdifferential of f at x, and it is denoted by ∂f (x).

Remark 2.2.1. [17, p. 716] x ∈ R p is a minimum of a convex function f if and only if

0 ∈ ∂f (x).
Example 2.2.2. [89, Lemma D.5] The subdifferential of the ℓ 1 norm at x ∈ R p is given by

∂∥.∥ 1 (x) = ∂|.|(x 1 ) × • • • × ∂|.|(x p ) where ∂|.|(t) =        {1} if t > 0 [-1, 1] if t = 0 {-1} if t < 0
The subdifferential of the ℓ ∞ norm at 0 is the unit ball of the ℓ 1 norm and for x ∈ R p where

x ̸ = 0 this subdifferential is equal to ∂∥.∥ ∞ (x) = s ∈ R p : ∥s∥ 1 = 1 and s i x i ≥ 0 if |x i | = ∥x∥ ∞ s i x i = 0 otherwise . Definition 2.

(Equality of patterns).

Let pen : R p → R be a convex penalizer. We say that x ∈ R p and z ∈ R p have the same pattern with respect to pen when ∂pen(x) = ∂pen(z), where ∂pen represents the subdifferential of pen.

We say that the estimator β recovers the pattern of β when

∂pen β = ∂pen(β).
Example 2.2.3 (LASSO pattern). In the LASSO regression, i.e., when pen(x) = ∥x∥ 1 , the pattern of β identifies with its sign vector:

patt(β) = sign(β).
The proof of the above example goes straightforwardly from Example 2.2.2.

Theorem 2.2.4 (Subdifferential description of the SLOPE pattern [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]).

Let Λ = (λ 1 , . . . , λ p ) satisfy λ 1 > . . . > λ p > 0. Then patt(b 1 ) = patt(b 2 ) ⇐⇒ ∂J Λ (b 1 ) = ∂J Λ (b 2 ).
The pattern of SLOPE defined in Definition 2.2.2 coincides with the following definition.

Definition 2.2.3 (SLOPE pattern [START_REF] Bogdan | Pattern recovery by SLOPE[END_REF]). The SLOPE pattern is a function

patt : R p → Z p such that patt(b) i = sign(b i )rank(|b i |),
where rank(|b i |) ∈ {1, 2, . . . , k} is defined to be the number of

|c j |'s satisfying |b i | ≥ |c j |, where |c 1 |, |c 2 |, . . . , |c k |, k ≤ p, are distinct non-zero values among |b 1 |, . . . , |b p |.
We adopt the convention that rank(0) = 0.

Fact 2.2.5 (Basic properties of SLOPE pattern [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]). (a) for every 1

≤ l ≤ ∥patt(b)∥ ∞ there exists j such that |patt(b) j | = l, (b) sign(patt(b)) = sign(b) (sign preservation), (c) |b i | = |b j | ⇒ |patt(b) i | = |patt(b) j | (cluster preservation), (d) |b i | > |b j | ⇒ |patt(b) i | > |patt(b) j | (hierarchy preservation). Example 2.2.6. patt(4, 0, -1.5, 1.5, -4) ′ = (2, 0, -1, 1, -2) ′ .
As a (nonzero) cluster in a vector M ∈ R p we denote a maximal (in terms of inclusion) set of indices i in supp(M ) such that the absolute value M i is the same. For example, the vector (3, -3, 1, 3, 1, 4) ′ has three clusters: {1, 2, 4}, {3, 5} and {6}.

Definition 2.2.4. Let M ̸ = 0 be a pattern in M p with k = ∥M ∥ ∞ nonzero clusters. The pattern matrix U M ∈ R p×k is defined as follows (U M ) ij = sign(m i )1 (|m i |=k+1-j) , i ∈ {1, . . . , p}, j ∈ {1, . . . , k}. Example 2.2.7. Let β = (3, -4, -3, 0, 4, 10) ′ . Then the pattern of β equals M = (1, -2, -1, 0, 2, 3) ′ and U M =          0 0 1 0 -1 0 0 0 -1 0 0 0 0 1 0 1 0 0         
.

Note that β can be represented by a product of U M and the vector (10, 4, 3) ′ ∈ R 3+ . For every β with pattern M such a vector in R 3+ exists.

Definition 2.2.5. Let M ̸ = 0 be a pattern in R p and k = max{∥M ∥ ∞ , 1}.

For X ∈ R n×p we define the clustered design matrix by X M = XU M ∈ R n×k and the clustered parameter by

Λ M = (U |M| ↓ ) ′ Λ. Example 2.2.8. Let X = (X 1 |X 2 |X 3 |X 4 ), M = (1, 2, -1, 0) ′ and Λ = (λ 1 , . . . , λ 4 ) ′ ∈ R 4+ . Then |M| ↓ = (2, 1, 1, 0) ′ .
The clustered matrix and the clustered parameter equal:

X M = (X 2 |X 1 -X 3 ) and Λ M = λ 1 λ 2 + λ 3 .

Convex Polytopes and cones

Convex analysis

We consider R n with the norm ∥ • ∥. Then the recession cone of C is defined as ∃ y 1 , . . . ,

Definition 2.3.1 (Affine set). [33, Sec. 2.1.2] A set C ∈ R n is affine if for any x 1 , x 2 ∈ C and θ ∈ R we have θx 1 + (1 -θ)x 2 ∈ C.

Definition 2.3.2 (Affine hull). [33, Sec. 2.1.2] The set of all affine combinations of points in

a set C ∈ R p is called the affine hull of C (aff(C)): aff(C) := {θ 1 x 1 + . . . θ k x k : x 1 , . . . , x k ∈ C, θ 1 , . . . , θ k ∈ R, θ 1 + . . . + θ k = 1}.
rb(C) := C \ riC. Definition 2.3.5 (Convex set). [33, Sec. 2.1.4] A set C ∈ R n is convex if for any x 1 , x 2 ∈ C and θ ∈ [0, 1] we have θx 1 + (1 -θ)x 2 ∈ C. Definition 2.3.6 (Convex hull). [33, Sec. 2.1.4] conv(C) := {θ 1 x 1 + . . . + θ k x k : x i ∈ C, θ i ≥ 0, i = 1, . . . , k, θ 1 + . . . + θ k = 1}. Remark 2.3.2. [33, Sec. 2.1.4] conv(C) is the smallest convex set containing C. Definition 2.3.7 (Convex cone). [33, Sec. 2.1.5] A set C ∈ R p is a convex cone if for any x 1 , x 2 ∈ C and θ 1 , θ 2 ≥ 0 we have θ 1 x 1 + θ 2 x 2 ∈ C. Definition 2.3.8 (Conic hull). [33, Sec. 2.1.5] cone(C) := {θ 1 x 1 + . . . + θ k x k : x i ∈ C, θ i ≥ 0, i = 1, . . . ,
rec(C) := {y ∈ R p : x + ty ∈ C for all x ∈ C, t ≥ 0}.
y l ∈ R p , α 1 , . . . , α l ∈ R A = l i=1 H(y i , α i ).
Then by -→ A we define the unique linear space parallel to A

- → A := l i=1 H(y i , 0).
Definition 2.3.14 (Polar set). [34, p. 37] For any M ⊂ R p , the polar set M * is defined by

M * := {y ∈ R p : ∀ x ∈ M : ⟨x, y⟩ ≤ 1}.
Equivalently,

M * = x ∈ M K(x, 1).

Polytopes

We recall basic definitions and facts about polytopes, which we will use throughout the proofs.

The following can be found in textbooks, such as [START_REF] Gruber | Convex and Discrete Geometry[END_REF] and [START_REF] Ziegler | Lectures on Polytopes[END_REF].

A set P ⊆ R p is called a polytope if it is the convex hull of a finite set of points {v 1 , . . . , v k } ⊆ R p , that is, P = conv{v 1 , . . . , v k }.
The dimension dim(P ) of a polytope is defined as the dimension of aff(P ), the affine subspace spanned by P . An inequality a ′ x ≤ c is called a valid inequality of

P if P ⊆ {x ∈ R p : a ′ x ≤ c}.
A face F of P is any subset F ⊆ P that satisfies

F = {x ∈ P : a ′ x = c}, where P ⊆ {x ∈ R p : a ′ x ≤ c},
for some a ∈ R p and c ∈ R. Note that F = ∅ and F = P are faces of P and that any face F is again a polytope. A non-empty face F with F ̸ = P is called proper. A point x 0 ∈ P lies in ri(P ) if x 0 is not contained in a proper face of P . We state two useful properties about faces in the following lemma.

Lemma 2.3.1. Let P ⊆ R p be a polytope given by

P = conv{v 1 , . . . , v k }, where v 1 , . . . , v k ∈ R p .
The following properties hold.

(a) If F and F are faces of P , then so is F ∩ F .

(b) Let L be an affine line contained in the affine hull of P . If L ∩ ri(P ) ̸ = ∅, then L intersects a proper face of P . (a) Let x ∈ R p and let pen be a polyhedral gauge. The pattern equivalence class C x is the set of all vectors having the same subdifferential as x:

Pattern equivalence class

C x := {w ∈ R p : ∂pen(w) = ∂pen(x)}. Definition 2.3.16. Let F ⊂ R p . As F ⊥ we denote the orthogonal complement of ----→ aff(F ).
Example 2.3.5. For any x ∈ R p , we have {x}

⊥ = {0} ⊥ = R p . Lemma 2.3.2. Let x 0 ∈ F . Then lin(F -x 0 ) = ----→ aff(F ).
Proof. By [101, Sec. III.5.

3.] we have

lin(R + (F -x 0 )) = aff(F ) -x 0 .
The statement is proved after observing that lin(F -

x 0 ) = lin(R + (F -x 0 )) and aff(F ) -x 0 = ----→ aff(F ).

Normal cones

Definition 2.3.17. [31, p. 15], [101, p.136] The normal cone to a convex set C at a point

x ∈ C, written N C (x) is the convex cone of normal vectors, i.e. vectors d in R p such that ⟨d, x -x⟩ ≤ 0 for all points x in C. Definition 2.3.18. [70, Def. 4.12.] If F is a face of a closed convex set K and x ∈ riF , then, N K (x) does not depend on x ∈ ri F and is denoted by N K (F ) and is called the cone of normals of K in F . Lemma 2.3.3. [70, Lemma 3.1.] Let K be a closed convex set in R n . To each x ∈ R n there exists a unique x ′ ∈ K such that ∥x -x ′ ∥ = inf y ∈ K ∥x -y∥.
Definition 2.3.19. [START_REF] Ewald | Combinatorial convexity and algebraic geometry[END_REF]Def. 3.2.] The map

p K : R n -→ K x → p K (x) = x ′ of Lemma 2.3.3 is called the nearest point map relative to K.
Definition 2.3.20. [START_REF] Ewald | Combinatorial convexity and algebraic geometry[END_REF]Def. 4.7.] Let x be a point of the closed convex set K. We call

N K (x) := -x + p -1 K (x)
the normal cone of K at x.

= p K (x + s). It implies that p -1 K (x) = x + N K (x), thus N K (x) = -x + p -1 K (x).
Corollary 2.3.1. The normal cone N K (x) is invariant under translations of the set K.

Theorem 2.3.6. [START_REF] Ewald | Combinatorial convexity and algebraic geometry[END_REF]Theorem 4.13.] Let K be a compact convex set in R p and x a relative

interior point of a face F ̸ = ∅ of K. Then {ri N K (x) : F is a face of K} = {ri N K (F ) : F is a face of K} is a partition (disjoint covering) of R p .
Definition 2.3.21 (Conjugate face). [34, p. 40] Let F be a face of a polytope P . The conjugate face F o of a dual polytope P * is defined as follows

F o := {y ∈ P * : ⟨x, y⟩ = 1 ∀ x ∈ F }.
Equivalently, we get

F o = x ∈ P K(x, 1) ∩ x ′ ∈ F H(x ′ , 1),
where the first and second intersection are equal respectively to P * and aff(F o ).

Consider a convex polytope P = {v ∈ R p : ⟨s j , v⟩ ≤ r j , j = 1, . . . , m}, cf. [101, p. 138]. We define its active set for x ∈ R p as A P (x) := {j = 1, . . . , m : ⟨s j , v⟩ = r j } Proposition 2.3.2. [START_REF] Gruber | Convex and Discrete Geometry[END_REF]Proposition 14.1] We have

N P (x) = cone({s j : j ∈ A P (x)}). Theorem 2.3.7. Let B * be the polar set of the polytope B. Let x ∈ ri(F ), where F is a face of B * . Then the normal cone at x to B * is given by N B * (x) = R + F o , where F o ⊂ B is the conjugate face to F . Proof. (R + F o ⊂ N B * (x)): Let y ∈ R + F o .
Then there exists such γ ≥ 0, that y ∈ γB and ⟨x, y⟩ = γ for every x ∈ F . It

implies that ⟨v, y⟩ ≤ γ ∀ v ∈ B * and ⟨x, y⟩ = γ ∀ x ∈ F.
Therefore ⟨v, y⟩ ≤ ⟨x, y⟩ for all v ∈ B * , i.e. ⟨v -x, y⟩ ≤ 0 for all v ∈ B * , which means that y ∈ N B * (x).

(

N B * (x) ⊂ R + F o ): Let y ∈ N B * (x), i.e. ⟨y, v -x⟩ ≤ 0 for every v ∈ B * . Thus the maximum of the value of ⟨y, v -x⟩ over v ∈ B * is attained at v = x.

Polyhedral gauges

Definition 2.3.22 (Gauge). [155, p. 53] Let K ⊂ R p be a closed convex set containing 0. The gauge function of K is defined as

inf{λ ≥ 0 : x ∈ λK}
The gauge function of K is also known as the Minkowski functional of K. When K is a polyhedron, its gauge function is known as a polyhedral gauge pen and can be written as the maximum of linear functions [START_REF] Rockafellar | Convex Analysis[END_REF][START_REF] Mousavi | Solution uniqueness of convex piecewise affine functions based optimization with applications to constrained ℓ 1 minimization[END_REF]:

∀x ∈ R p , pen(x) = max{0, u ′ 1 x, . . . , u ′
l x}, for some u 1 , . . . , u l ∈ R p . Note that a polyhedral gauge, whose unit ball {x ∈ R p : pen(x) ≤ 1} is a bounded and symmetric with respect to the origin polyhedron, is a polyhedral norm. Let pen be a polyhedral gauge,

Thresholded penalized least squares estimation

X ∈ R n×p , Y ∈ R n and λ > 0. Given β ∈ S X,λpen (Y ), we say that β is a thresholded estimator of β if ∂pen( β) ⊆ ∂pen( β).
Example 2.3.8. In SLOPE optimization problem, a vector β = (5, 5, 5, 0, 0) ′ is a thresholded estimator of βSLOP E = (4, 4, 2, 0, 0) ′ .

Permutahedron

Definition 2.3.24. Let Λ ∈ R p . A permutahedron P Λ is a convex hull of all possible permutations of Λ:

P Λ := Conv((λ π(1) , . . . , λ π(p) ) ′ : π ∈ S p ). (2.3.1) Definition 2.3.25 (Signed permutahedron). Let Λ ∈ R p . A signed permutahedron C Λ is a
convex hull of all possible permutations and sign changes of Λ:

C Λ = {π = (π 1 , π 2 , . . . , π p ) ∈ R p : j≤i |π| (j) ≤ j≤i λ j : i = 1, 2, . . . , p}. (2.3.2)
C Λ is a unit ball in a norm J * Λ dual to the SLOPE norm J Λ .

Basics on Moore-Penrose inverse

The notion of the Moore-Penrose inverse is crucial in the SLOPE irrepresentability condition and is outlined below, see [START_REF] Golub | Matrix computations[END_REF][START_REF] Ben-Israel | Generalized inverses, volume 15 of CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC[END_REF].

If A is an n × p real matrix then a p × n matrix A + is called a Moore-Penrose inverse of A if AA + A = A, A + AA + = A +
and if the matrices AA + and A + A are symmetric.

There always exists a unique real Moore-Penrose inverse A + of a real matrix A. In some cases it may be computed quickly to a more convenient form:

• If A ′ A is an invertible matrix, then A + = (A ′ A) -1 A ′ , • If AA ′ is an invertible matrix, then A + = A ′ (AA ′ ) -1 , • A + = A ′ (AA ′ ) + , • 0 + = 0.
In general, the Moore-Penrose inverse A + is computed using the formula A + = A ′ (AA ′ ) + and the Singular Value Decomposition, which allows to do it numerically.

Remark 2.3.9.

• AA + is an orthogonal projector onto col(A),

• col(A + ) = col(A ′ ).

Functional analysis

∥x∥ * = sup{x ′ b : ∥b∥ ≤ 1}.
In particular, the dual sorted ℓ 1 norm J Λ has an explicit expression given in [START_REF] Negrinho | Orbit regularization[END_REF]:

J * Λ (b) = max |b| (1) λ 1 , 2 i=1 |b| (i) 2 i=1 λ i , . . . , p i=1 |b| (i) p i=1 λ i , b ∈ R p .

Tools from optimization

S α = {x : f (x) ≤ α}, for α ∈ R, are convex. If a function -f is quasiconvex, then
x→x 0 f (x) ≥ f (x 0 )
for all x 0 ∈ R p . If a function -f is lower semi-continuous, then we say that f is upper semicontinuous.

Theorem 2.3.10 (max-min inequality). [START_REF] Boyd | Convex Optimization[END_REF]Sec. 5.4.1]

sup b ∈ B inf a ∈ A f (a, b) ≤ inf a ∈ A sup b ∈ B f (a, b) Definition 2.3.29 (Saddle point). [33, Sec. 5.4.2] The pair (a, b) ∈ A × B is a saddle point for a function f (and A and B) if f (a, b) ≤ f (a, b) ≤ f (a, b)
for all a ∈ A and b ∈ B. In other words, we have

f (a, b) = inf a ∈ A f (a, b) = sup b ∈ B f (a, b).
The existence of the saddle point implies that it attains the equality in the max-min inequality: Theorem 2.3.12 (Hardy-Littlewood-Pólya rearrangement inequality). [START_REF] Hardy | Inequalities[END_REF]Theorem 368] For

sup b ∈ B inf a ∈ A f (a, b) = inf a ∈ A sup b ∈ B f (a, b) = f (a, b).
every x 1 ≤ x 2 ≤ . . . ≤ x k ∈ R, y 1 ≤ y 2 ≤ . . . ≤ y k ∈ R
and for every permutation σ ∈ S k we have

x k y 1 + x k-1 y 2 + . . . + x 1 y k ≤ x σ(1) y 1 + x σ(2) y 2 . . . x σ(k) y k ≤ x 1 y 1 + x 2 y 2 + . . . + x k y k .

Tools from probability

sup n n i=1 a 2 ni σ 2 n < ∞ and max 1≤i≤n a ni σ n → 0 as n → ∞, (2.3.3)
where

σ 2 = V ar n i=1 a ni ξ i . If j:|i-j|≥u Cov(ξ i , ξ j ) -→ 0 as u → ∞ uniformly in i ≥ 1, then, as n → ∞, 1 σ n n i=1 a ni ξ i d → N (0, 1). Theorem 2.3.17 (Tail inequality). If X ∼ N (0, 1), then P(X > t) ≤ 1 t e -t 2 /2 1 √ 2π .
Proof. [43] Note that for x ≥ t > 0 we have x t ≥ 1. Therefore

P(X ≥ t) = 1 √ 2π ∞ t 1 • e -x 2 /2 dx ≤ 1 √ 2π ∞ t x t • e -x 2 /2 dx = 1 t √ 2π e -t
for some r > 2. Let {a ni } be a double array of constants satisfying ∞ -∞ a 2 ni < ∞ for every n. Define S n = ∞ -∞ a ni ε n . (2.3.5)
Assume that as n → ∞,

A n = ∞ -∞ a 2 ni → ∞,
and sup i a 2 ni = o(A n (log A n )) -ρ for all ρ > 0. (2.3.6) If there exist constants c i ≥ 0 and d > 2/r such that ∥a n -a m ∥ 2 ≤   n i=m+1 c i   d for n > m > m 0 (2.3.7) and   n i=m 0 c i   d = O(A n ) a.s. n → ∞, (2.3.8) then lim sup n→∞ |S n | (2A n log log A n ) 1/2 ≤ σ a.s.

Chapter 3

Pattern recovery and signal denoising by SLOPE when the design matrix is orthogonal

Introduction

Introduction and motivations

The content of this chapter may be found in the recently published article of the author of the dissertation, Graczyk, Kołodziejek and Wilczyński [START_REF] Skalski | Pattern recovery and signal denoising by slope when the design matrix is orthogonal[END_REF]. To start our discussion on the pattern recovery by penalized linear regression methods, we start with the SLOPE estimator and the simplest case of the design matrix X being orthogonal. For that reason let us recall the Linear Multiple Regression. It concerns the model Y = Xβ + ε, where Y ∈ R n is an output vector, X ∈ R n×p is a fixed design matrix, β ∈ R p is an unknown vector of predictors and ε ∈ R n is a noise vector. The primary goal is to estimate β. In the low-dimensional setting, that is, when the number of predictors p is not greater than the number of explanatory variables n and X is of full rank, the ordinary least squares estimator βOLS has an exact formula βOLS = (X ′ X) -1 X ′ Y . For practical reasons there is an urge to avoid the high-dimensionality curse, therefore we want the estimate to be sparse, i.e., to be descriptible by a smaller number of parameters. Several solutions were proposed to deal with such problem. One of them, the Least Absolute Shrinkage and Selection Operator (LASSO [START_REF] Chen | Basis pursuit[END_REF][START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF]) involves penalizing the residual sum of squares ∥Y -X β∥ 2 2 with an ℓ 1 norm of β multiplied by a tuning parameter λ:

βLASSO := arg min b ∈ R p 1 2 ∥Y -Xb∥ 2 2 + λ∥b∥ 1 .
We turn our focus on one of the extensions of LASSO, which is the Sorted ℓ 1 Penalized Estimator (SLOPE [START_REF] Bogdan | Statistical estimation and testing via the sorted l1 norm[END_REF][START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF][START_REF] Zeng | Decreasing weighted sorted ℓ 1 regularization[END_REF]). In addition, SLOPE allows one to clusterize the similar coefficients of β. In SLOPE, the ℓ 1 norm is replaced by its sorted version J Λ , which depends on the tuning vector Λ = (λ 1 , . . . , λ p ) ∈ R p , where λ 1 ≥ . . . ≥ λ p ≥ 0:

J Λ (β) := p i=1 λ i |β| (i) ,
where {|β| (i) } p i=1 is a decreasing permutation of absolute values of β 1 , . . . , β p :

βSLOPE := arg min b ∈ R p 1 2 ∥Y -Xb∥ 2 2 + J Λ (b) .
25

The case of Λ being an arithmetic sequence was studied by Bondell and Reich [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF] and called the Octagonal Shrinkage and Clustering Algorithm for Regression (OSCAR). The special case of SLOPE with λ 1 = λ 2 = . . . = λ p > 0 is LASSO. For Λ = (0, . . . , 0) we obtain the OLS estimator.

Clustering the predictors allows for additional dimension reduction by identifying variables with the same absolute values of the regression coefficients. Recently, interest has increased in methods that cluster highly correlated predictors [START_REF] Bondell | Simultaneous factor selection and collapsing levels in anova[END_REF][START_REF] Gertheiss | Sparse modeling of categorial explanatory variables[END_REF][START_REF] Maj-Kańska | Delete or merge regressors for linear model selection[END_REF][START_REF] Nowakowski | Group lasso merger for sparse prediction with high-dimensional categorical data[END_REF][START_REF] Oelker | Regularization and model selection with categorical predictors and effect modifiers in generalized linear models[END_REF][START_REF] Stokell | Modelling high-dimensional categorical data using nonconvex fusion penalties[END_REF]. SLOPE is ideal for this task, since it is capable to identify the low-dimensional structure, which is called the SLOPE pattern, defined by Schneider and Tardivel with the subdifferential of the SLOPE norm J Λ , see Theorem 2.2.4. The clustering properties of SLOPE have been studied before, cf. [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF][START_REF] Figueiredo | Ordered weighted l1 regularized regression with strongly correlated covariates: Theoretical aspects[END_REF], but the researchers consider strongly correlated predictors, which are used in financial mathematics to group the assets with respect to their partial correlation with hedge fund return times series [START_REF] Kremer | Sparse index clones via the sorted ℓ 1 -norm[END_REF]. We decided to consider the pattern recovery in more general case. We start with supposing the orthogonal design

X ′ X = nI p . (3.1.1)
This is a classical and natural assumption in the case of experimental data; cf. [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF]. Moreover, in the asymptotic case, where n → ∞ and p is fixed, it is usually supposed that X ′ X/n → C > 0, cf. [START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF]. In (3.1.1) the design matrix X is orthogonal. Then, the Euclidean norm of each n-dimensional column of X equals n. If it was 1, the terms of X would approach zero for large n, which is not natural. This class of matrices is being widely used in signal analysis, [START_REF] Rao | Orthogonal transforms for digital signal processing[END_REF][START_REF] Chen | Atomic decomposition by basis pursuit[END_REF].

For general X the problem is considered in Chapter 4.

To study the properties of SLOPE we often use the closed unit ball C Λ in the dual norm of J Λ , which was studied, for example, by Zeng and Figueiredo [START_REF] Zeng | Decreasing weighted sorted ℓ 1 regularization[END_REF]. This dual ball is described explicitely as a signed permutahedron, see, e.g. [START_REF] Negrinho | Orbit regularization[END_REF][START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]:

C Λ =    π = (π 1 , π 2 , . . . , π p ) ∈ R p : j≤i |π| (j) ≤ j≤i λ j : i = 1, 2, . . . , p    . (3.1.2)
Here we prove novel results on the strong consistency of SLOPE both in estimation and in pattern recovery. We also introduce a new method, based on the minimax approach, to find the relationship between βSLOPE and βOLS .

Outline

In Section 3.2 we derive the connections between βSLOPE and βOLS in the orthogonal design. We use the minimax theorem of Sion, cf. [START_REF] Aubin | Mathematical methods of game and economic theory[END_REF]. In Section 3.3 we focus on the properties of βSLOPE . We use the geometric interpretation of SLOPE to explain its ability to identify the SLOPE pattern and provide new theoretical results on support recovery and clustering properties using a representation of SLOPE as a function of the ordinary least squares (OLS) estimator. A similar approach for LASSO was used by Ewald and Schneider, cf. [START_REF] Ewald | Uniformly valid confidence sets based on the Lasso[END_REF].

To analyze the asymptotic properties of the SLOPE estimator, e.g., its consistency, we have to assume that the sample size n tends to infinity. Therefore, in Section 3.4 we define a sequence of linear regression models

Y (n) = X (n) β + ε (n) n .
In this sequence, the response vector Y (n) ∈ R n , the design matrix X (n) ∈ R n×p and the error term

ε (n) = (ε (n) 1 , ε (n) 2 , . . . , ε (n) n ) ′ ∈ R n varies with n and has the normal distribution N (0, σ 2 I n ).
We make no assumptions about the relations between ε (n) and ε (m) for n ̸ = m.

In this chapter we consider the specific, but statistically important, model in which n ≥ p and the columns of X are orthogonal. The orthogonality assumption allows us to derive, by simple techniques, relatively precise results on the SLOPE estimator (e.g., Theorem 3.1), which seem unavailable when columns of X are not orthogonal. We provide the conditions under which the SLOPE estimator is strongly consistent. Additionally, in case when for each n the design matrix is orthogonal, we provide the conditions on the sequence of tuning parameters such that SLOPE is strongly consistent in the pattern recovery. In Section 3.5 we show the applications of the SLOPE clustering in terms of high frequency signal denoising and illustrate them with simulations. The Appendix covers the proofs of technical results. Substantially more difficult techniques based on subdifferential calculus are developed in [START_REF] Bogdan | Pattern recovery by SLOPE[END_REF]. These techniques are used in [START_REF] Bogdan | Pattern recovery by SLOPE[END_REF] to establish the properties of the SLOPE estimator in the general case, where the columns of X are not orthogonal and p may be much larger than n. However, the asymptotic results of [START_REF] Bogdan | Pattern recovery by SLOPE[END_REF] are derived under stronger assumptions than those of this paper. In [START_REF] Bogdan | Pattern recovery by SLOPE[END_REF] the sequence of error terms ε (n) is incremental and the sequence of tuning parameters has the form λ n = α n Λ, where α n is a given sequence of positive numbers and Λ ∈ R p is fixed. In this paper we make no assumptions about the relations between ε (n) and ε (m) for n ̸ = m, and the sequence of tuning parameters has a general form.

Approach by minimax theorem

Technical results

Let X ∈ R n×p be a real-valued matrix. Let r SLOPE denote the minimum value of the SLOPE criterion, attained by βSLOPE , i.e.

r SLOPE := min b ∈ R p 1 2 ∥Y -Xb∥ 2 2 + J Λ (b) = 1 2 ∥Y -X β SLOPE ∥ 2 2 + J Λ ( β SLOPE ). Since ∥ β SLOPE ∥ 2 ≤ √ p∥ β SLOPE ∥ ∞ and λ 1 ∥ β SLOPE ∥ ∞ ≤ J Λ ( β SLOPE ) ≤ r SLOPE , it follows that λ 1 β SLOPE 2 ≤ √ p r SLOPE ≤ √ p 1 2 ∥Y -X0∥ 2 2 + J Λ (0) = √ p 2 ∥Y ∥ 2 2 .
We immediately get the following result.

Corollary 3.2.1. β SLOPE 2 2 ≤ M 0 , where M 0 = p ∥Y ∥ 4 2 4λ 2 1 .
From this corollary it is seen that we can clearly limit our search to vectors β from the compact set M ⊂ R p defined by M := b ∈ R p : ∥b∥ 2 2 ≤ M 0 . Therefore, we can equivalently define a SLOPE solution by

β SLOPE = arg min b ∈ M 1 2 ∥Y -Xb∥ 2 2 + J Λ (b) . (3.
J Λ (b) = max π∈C Λ π ′ b. (3.2.2)
The proof is a simple application of the definition of the dual norm and the reflexivity of (R p , J Λ ) = (R p , J * Λ ) * . Thus

J Λ (b) = ∥b∥ (R p ,J Λ ) = sup x:J * Λ (x)≤1
x ′ b.

Saddle point

In this section we continue the assumption that X ∈ R n×p is a real-valued matrix. Let the function r : M × C Λ → R be defined by

r(b, π) := 1 2 ∥Y -Xb∥ 2 2 + π ′ b.
As an immediate consequence of (3.2.1) and Proposition 3.2.1 we obtain

r SLOPE = min b ∈ R p 1 2 ∥Y -Xb∥ 2 2 + J Λ (b) = min b ∈ M 1 2 ∥Y -Xb∥ 2 2 + J Λ (b) = min b ∈ M max π∈C Λ 1 2 ∥Y -Xb∥ 2 2 + π ′ b = min b ∈ M max π∈C Λ r(b, π).
It turns out that the order of maximization over π ∈ C Λ and minimization over b ∈ M can be switched without affecting the result. To see this, note that both C Λ and M are convex and compact. Furthermore, for each fixed π ∈ C Λ , r(b, π) is a convex continuous function with respect to b ∈ M and, for each fixed b ∈ M, r(b, π) is concave with respect to π ∈ C Λ (in fact, it is linear). Therefore, all assumptions of the Sion's minimax theorem are fulfilled (see [7, p. 218]) and thus there exists a saddle point (β

* , π * ) ∈ M × C Λ such that max π∈C Λ min b ∈ M r(b, π) = min b ∈ M r(b, π * ) = r(β * , π * ) = max π∈C Λ r(β * , π) = min b ∈ M max π∈C Λ r(b, π) = r SLOPE .
In the next section we shall see that the first coordinate of any saddle point (β * , π * ) is the SLOPE estimator.

SLOPE solution when X has full column rank

r(b, π) = 1 2 Y ′ Y -Y ′ Xb + 1 2 b ′ X ′ Xb + π ′ b
and differentiate with respect to b, we obtain

∂r(b, π) ∂b = -X ′ (Y -Xb) + π.
Equating this gradient with 0 gives the following equation for the optimum point b π :

X ′ Xb π = X ′ Y -π. (3.2.3)
Substituting this into the equation for r(b π , π) and using the fact that (X ′ X) -1 exists, we find that

r(b π , π) = 1 2 Y ′ Y -b ′ π X ′ Y + 1 2 b ′ π X ′ Xb π + π ′ b π = 1 2 Y ′ Y -b ′ π X ′ Y + b ′ π X ′ Xb π + b ′ π π - 1 2 b ′ π X ′ Xb π = 1 2 Y ′ Y - 1 2 b ′ π X ′ Xb π = 1 2 Y ′ Y - 1 2 b ′ π X ′ X(X ′ X) -1 X ′ Xb π = 1 2 Y ′ Y - 1 2 (X ′ Y -π) ′ (X ′ X) -1 (X ′ Y -π). Let p j = |{i : |m i | = k + 1 -
= (π * 1 , . . . , π * p ) ′ ∈ C Λ be any solution of π * = arg min π∈C Λ (X ′ Y -π) ′ (X ′ X) -1 (X ′ Y -π)
and let β * = (β * 1 , . . . , β * p ) ′ be the corresponding point from M given by

β * = (X ′ X) -1 (X ′ Y -π * ).
Then, (ππ * ) ′ β * ≤ 0, for all π ∈ C Λ and hence

(a) sign (β * i ) • sign (π * i ) ≥ 0, i = 1,
) i | = k + 1 -j, then |π * | i ∈ |π * | (P j-1 +1) , . . . , |π * | (P j ) , (c) for any permutation τ satisfying |β * τ (1) | ≥ . . . ≥ |β * τ (p) |, if there is a k ∈ {2, . . . , p} such that k-1 i=1 π * τ (i) < k-1 i=1 λ i and π * τ (k) > 0, then β * τ (k-1) = β * τ (k) .
The proof is given in the Appendix. An immediate consequence of the Lemma is the following result.

Lemma 3.2.2. Assume that X has full column rank. The point (β * , π * ) defined as in Lemma 3.2.1 is the saddle point of the function r(b, π).

The proof is given in the Appendix. We use the last lemma to prove the main result of this section.

π∈C Λ r(β * , π) = min b ∈ M max π∈C Λ r(b, π) (see previous lemma) we have 1 2 ∥Y -Xβ * ∥ 2 2 + J Λ (β * ) = max π∈C Λ 1 2 ∥Y -Xβ * ∥ 2 2 + π ′ β * = max π∈C Λ r(β * , π) = min b ∈ M max π∈C Λ r(b, π) = min b ∈ R p 1 2 ∥Y -Xb∥ 2 2 + J Λ (b) .
Corollary 3.2.2. In the linear model satisfying 1 n X ′ X = I p we have

βOLS -βSLOPE = 1 n π * = 1 n arg min π∈C Λ βOLS -π 2 2 = arg min π∈C Λ/n βOLS -π 2 2 ,
is the proximal projection of βOLS onto C Λ/n .

Projections onto C Λ are widely used in [START_REF] Minami | Degrees of freedom in submodular regularization: A computational perspective of Stein's unbiased risk estimate[END_REF] in the study of the notion of degrees of freedom. However, the Corollary 3.2.2 is not stated there explicitely.

Remark 3.2.3. Assume that X has full column rank. For each π ∈ C Λ , the point b π defined in (3.2.3) belongs to b ∈ R p : ∥b∥ 2 2 ≤ M ,
where M is chosen so that M > max{M 0 , M 1 } with

M 1 := max π∈C Λ ∥(X ′ X) -1 (X ′ Y -π)∥ 2 2 ≤ M.

Properties of SLOPE in the orthogonal design

SLOPE vs. OLS

By Theorem 3.2.2 and Corollary 3.2.2, when 1 n X ′ X = I p , the orthogonal projection of the ordinary least squares estimator βOLS = 1 n X ′ Y onto the unit ball C Λ/n is equal to βOLS -βSLOPE . For Λ = (200, 100) ′ and n = 50 this property is illustrated in Figure 3.1. The figure represents βSLOPE (black arrows) depending on the localization of βOLS in the orthogonal design. For βOLS being the blue point located in the area labeled by (1, 0) the first component of βSLOPE is positive and the second is null. For βOLS being the yellow point located on the area labeled by (-1, 1) both components of βSLOPE have equal absolute value (clusterization), but their signs are opposite. For βOLS being the red point located on the area labeled by (1, 2), both components of βSLOPE are positive and the first component is smaller than the second one. The blue polytope is the dual SLOPE unit ball C Λ and labels M 2 = {(0, 0), (±1, 0), (0, ±1), (±1, ±1), (±2, ±1), (±1, ±2)} associated to the areas of this figure correspond to all SLOPE patterns for n = 50 and p = 2. In the orthogonal design, one may also explicitly compute the SLOPE estimator. Indeed, by the Corollary 3.2.2, βSLOPE is the image of βOLS by the proximal operator of the SLOPE norm. Therefore, this operator has a closed form formula [START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF][START_REF] Tardivel | Simple expressions of the lasso and slope estimators in low-dimension[END_REF][START_REF] Dupuis | Proximal operator for the sorted ℓ 1 norm: Application to testing procedures based on SLOPE[END_REF]. This explicit expression gives an analytical way to learn that the SLOPE solution is sparse and built of clusters.

β 1 β 2 π * SLOPE OLS π * SLOPE OLS π * SLOPE OLS (-2,1) (-1,2) (1,2) (2,1)
(2,-1)

(1,-2) (-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1) (1,0)
(1,-1) (0,-1) (-1,-1) (0,0)

∈ R p 1 2n ∥Y -Xb∥ 2 2 + J Λ (b) = arg min b ∈ R p 1 2 β OLS -b 2 2 + J Λ (b) . (3.3.1)
Our proof of this Lemma is given in the Appendix. The next theorem gives a sufficient condition for the clustering effect of the SLOPE estimator in the orthogonal design.

Theorem 3.3.1. Consider a linear model with orthogonal design 1 n X ′ X = I p . Let π be a permutation of (1, 2, . . . , p) such that

β OLS π(1) ≥ β OLS π(2) ≥ . . . ≥ β OLS π(p) . For i ∈ {1, 2, . . . , p -1}, if β OLS π(i) -β OLS π(i+1) ≤ λ i -λ i+1 n , then β SLOPE π(i) = β SLOPE π(i+1) . Proof. By Lemma 3.3.1, in the orthogonal design, β SLOPE is the proximal map of J Λ/n (•)
at βOLS . The result may be inferred from [START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF]Lemma 2.3].

In the following theorem we derive the necessary and sufficient conditions under which SLOPE in the orthogonal design recovers the support of the vector

β = (β 1 , . . . , β p ) ′ , i.e. β SLOPE i = 0 ⇐⇒ β i = 0. Theorem 3.3.2. Under orthogonal design 1 n X ′ X = I p , let π be a permutation of (1, 2, . . . , p) that satisfies β OLS π(1) ≥ | β OLS π(2) | ≥ . . . ≥ | β OLS π(p) |.
Without loss of generality suppose that supp(β) = {1, 2, . . . , p 0 } with p 0 < p. The necessary and sufficient condition for SLOPE to identify the set of relevant covariables is:

(a) min 1≤i≤p 0 β OLS i > max p 0 +1≤i≤p β OLS i , (b) p 0 i=k β OLS π(i) > 1 n p 0 i=k λ i , for k = 1, 2, . . . , p 0 , (c) k i=p 0 +1 β OLS π(i) ≤ 1 n k i=p 0 +1 λ i , for k = p 0 + 1, p 0 + 2, . . . , p.
Proof. The result may be inferred from the properties of the proximal SLOPE [27, Lemma 2.3 and Lemma 2.4] and from Lemma 3.3.1.

Asymptotic properties of SLOPE

In this section we discuss several asymptotic properties of SLOPE estimators in the lowdimensional regression model in which p is fixed and the sample size n tends to infinity. For each n ≥ 1 we consider a linear model

Y (n) = X (n) β + ε (n) , (3.4.1)
where

Y (n) = (y (n) 1 , y (n) 2 , . . . , y (n) n ) ′ ∈ R n is a vector of observations, X (n) ∈ R n×p is a deter- ministic design matrix with rank(X (n) ) = p, β = (β 1 , β 2 , . . . , β p ) ′ ∈ R p is a vector of unknown regression coefficients and ε (n) = (ε (n) 1 , ε (n) 2 , . . . , ε (n) n ) ′ ∈ R n
is a noise term, which has the normal distribution N (0, σ 2 I n ). We make no assumptions about the dependence between ε (n) and ε (m) for n ̸ = m. In particular, ε (n) does not need to be a subsequence of ε (m) . When defining the sequence ( β SLOPE n

) of SLOPE estimators, we assume that the tuning vector varies with n. More precisely, for each n ≥ 1 its coefficients λ

(n) 1 ≥ λ (n) 2 ≥ . . . ≥ λ (n) p ≥ 0 are fixed and λ (n) 1 > 0. By β SLOPE n
we denote the SLOPE estimator corresponding to the tuning vector

Λ (n) = (λ (n) 1 , . . . , λ (n) p ) ′ : β SLOPE n = arg min b ∈ R p 1 2 Y (n) -X (n) b 2 2 + J Λ (n) (b) . (3.4.2)

Strong consistency of the SLOPE estimator

n n -1 X (n) ′ X (n) = C,
where C is a positive definite matrix. Let β SLOPE n

, n ≥ 1, be the SLOPE estimator corresponding to the tuning vector

Λ (n) = (λ (n) 1 , λ (n) 2 , . . . , λ (n) p ) ′ . (a) If lim n→∞ λ (n) 1 n = 0, then β SLOPE n a.s. -→ β. (b) If lim n→∞ λ (n) 1 n = λ 0 > 0
If Z ∼ N(0, 1), then P(Z > t) ≤ t -1 e -t 2 /2 / √ 2π, t > 0. Lemma 3.4.1. Assume that (Q n )
n∈N is a sequence of Gaussian random variables, defined on the same probability space, which converges in distribution to N(0, σ 2 ) for some σ ∈ (0, ∞).

Then, for any δ > 0,

lim n→∞ Q n (log(n)) 1/2+δ = 0 a.s.
Our proof of the strong consistency of SLOPE is based on the strong consistency of the OLS estimator. The latter result is a folklore and we prove it in our setting.

If lim

n n -1 (X (n) ) ′ X (n) = C, where C is positive definite, then β OLS n a.s. -→ β.
Proof. We have

β OLS n -β = ((X (n) ) ′ X (n) ) -1 (X (n) ) ′ Y (n) -β = ((X (n) ) ′ X (n) ) -1 (X (n) ) ′ ε (n) . Then √ n β OLS n -β has the normal distribution N (0, σ 2 (n -1 (X (n) ) ′ X (n) ) -1
) and its components satisfy the assumptions of Lemma 3.4.1. Since log(n

) 1/2+δ = o( √ n),
* n ∈ C (Λ (n) ) such that β SLOPE n = ((X (n) ) ′ X (n) ) -1 ((X (n) ) ′ Y (n) -π * n ). Since π * n takes values in C Λ (n) , it follows that ∥π * n ∥ ∞ ≤ λ (n) 1 . Hence, π * n n a.s. -→ 0, (3.4.3) because π * n n ∞ ≤ λ (n) 1 n → 0.
The assumption that rank(X (n) ) = p implies that the matrix (X (n) ) ′ X (n) is invertible and therefore the least squares estimator of β is unique and has the form

β OLS n = ((X (n) ) ′ X (n) ) -1 (X (n) ) ′ Y (n) . Combining with (3.4.3) the fact that β OLS n a.s.
-→ β, we conclude that

β SLOPE n = ((X (n) ) ′ X (n) ) -1 ((X (n) ) ′ Y (n) -π * n ) = β OLS n -((X (n) ) ′ X (n) ) -1 π * n = β OLS n - (X (n) ) ′ X (n) n -1 π * n n a.s. -→ β -C -1 0 = β. (b) Since β SLOPE n minimizes over b ∈ R p the function l(b) := 1 2 ∥Y (n) -X (n) b∥ 2 2 + J Λ (n) (b)
and since λ

(n) 1 ∥b∥ ∞ ≤ J Λ (n) (b), it follows that 0 ≤ l(0) -l( β SLOPE n ) = ( β SLOPE n ) ′ (X (n) ) ′ Y (n) - 1 2 ( β SLOPE n ) ′ (X (n) ) ′ X (n) β SLOPE n -J Λ (n) ( β SLOPE n ) ≤ ( β SLOPE n ) ′ (X (n) ) ′ Y (n) - 1 2 ( β SLOPE n ) ′ (X (n) ) ′ X (n) β SLOPE n -λ (n) 1 ∥ β SLOPE n ∥ ∞ = ( β SLOPE n ) ′ (X (n) ) ′ X (n) β OLS n - 1 2 ( β SLOPE n ) ′ (X (n) ) ′ X (n) β SLOPE n -λ (n) 1 ∥ β SLOPE n ∥ ∞ .
The last equality follows from the fact that (X

(n) ) ′ Y (n) = (X (n) ) ′ X (n) β OLS n .
Suppose to the contrary that the true parameter β satisfies λ 0 ∥β∥ ∞ > β ′ Cβ/2 (which is the case when ∥β∥ ∞ is sufficiently close to 0) and that β SLOPE n a.s.

-→ β. Then, using the facts that β OLS n a.s. -→ β when β satisfies λ 0 ∥β∥ ∞ ≤ β ′ Cβ/2.

-→ β and that lim

n n -1 (X (n) ) ′ X (n) = C, we have 0 ≤ l(0) -l( β SLOPE n ) n a.s. -→ β ′ Cβ - 1 2 β ′ Cβ -λ 0 ∥β∥ ∞ = 1 2 β ′ Cβ -λ 0 ∥β∥ ∞ ,

Asymptotic pattern recovery in the orthogonal design

We again consider a sequence of linear models (3.4.1) but this time we assume that for each n the deterministic design matrix X (n)

of size n × p satisfies (X (n) ) ′ X (n) = nI p . (3.4.4)
As usual, we assume Gaussian errors

ε (n) ∼ N(0, σ 2 I n ).
Let

β SLOPE n = β SLOPE 1 (n), . . . ,
lim n→∞ λ (n) 1 n = 0
and that there exists δ > 0 such that

lim inf n→∞ λ (n) i -λ (n) i+1 √ n (log(n)) 1/2+δ = m > 0 for i = 1, . . . , p -1. ( 3

→ patt(β).

Note that above conditions are satisfied e.g. by λ

(n) i = c(p + 1 -i)n 2/3 for any constant c > 0.
Proof. Without loss of generality we may assume that β = (β 1 , . . . , β p ) ′ and β 1 ≥ β 2 ≥ . . . ≥ β p ≥ 0. Indeed, we can always achieve such condition by permuting the columns of X (n) and changing their signs. Since the space of patterns is discrete, we have to show that for large n, patt( βSLOPE n ) = patt(β) a.s. We divide the proof into the following four parts:

(a)

β i = β j > 0 =⇒ β SLOPE i (n) = β SLOPE j (n) a.s. for large n, (b) β i > β i+1 =⇒ β SLOPE i (n) > β SLOPE i+1 (n) a.s. for large n, (c) β i = 0 =⇒ β SLOPE i (n) = 0 a.s. for large n, (d) β i > 0 =⇒ β SLOPE i (n) > 0 a.s. for large n.
The points (b) and (d) follow quickly by the strong consistency of βSLOPE (n). To prove (a) and (c) we observe that for each n we are in the orthogonal design case. Let π n be a permutation of (1, 2, . . . , p) satisfying

| β OLS πn(1) (n)| ≥ | β OLS πn(2) (n)| ≥ . . . ≥ | β OLS πn(p) (n)|.
By the strong consistency of the OLS estimator, taking n sufficiently large, we may ensure that the clusters of β do not interlace in β OLS n in the sense that if

β i > β j , then β OLS i (n) > β OLS j
(n) a.s. for n sufficiently large. Let us now consider point (a). Let S i denote the cluster containing β i > 0, that is, the set S i = {j ∈ {1, . . . , p} : β j = β i }. In view of the ordering of β, there exists k i ∈ {1, . . . , p} such that

S i = {π n (j) : j ∈ {k i , k i + 1, . . . , k i + #S i -1}} .
We will show that if π n (k), π n (k + 1) ∈ S i , then for large n

β SLOPE πn(k) (n) = β SLOPE πn(k+1) (n) a.s., (3.4.6) thus β SLOPE j (n) = β SLOPE k (n) for j, k ∈ S i
, which finishes the proof of (a). Now assume that π n (k), π n (k+1) ∈ S i . Then, by Theorem 3.3.1, the condition (3.4.6) is satisfied if

β OLS πn(k) (n) -β OLS πn(k+1) (n) ≤ 1 n λ (n) k -λ (n) k+1 (3.4.7)
holds for large n and both β OLS πn(k) (n) and β OLS πn(k) (n) have the same sign. The latter is ensured by the strong consistency of the OLS estimator and the fact that β i > 0. If π n (k), π n (k + 1) ∈ S i , then we have the following bound

β OLS πn(k) (n) -β OLS πn(k+1) (n) ≤ j∈S i β OLS j (n) -β OLS i (n) . ( 3
p 0 i=k λ (n) i ≤ p 0 λ (n) 1 n ,
which converges to 0. On the other hand, the left-hand side of (b) converges a.s. to p 0 i=k β i , which is positive. Thus, condition (b) from Theorem 3.3.2 holds for large n. Condition (c) from Theorem 3.3.2 follows from Lemma 3.4.1. Indeed, we have for δ > 0 and k = p 0 + 1, . . . , p,

lim n→∞ √ n (log(n)) 1/2+δ k i=p 0 +1 | β OLS πn(i) (n)| = k i=p 0 +1 lim n→∞ | √ n β OLS πn(i) (n)| (log(n)) 1/2+δ = 0 a.s., while lim n→∞ 1 √ n(log(n)) 1/2+δ k i=p 0 +1 λ (n) i ≥ k i=p 0 +1 lim n→∞ λ (n) i -λ (n) i+1 √ n(log(n)) 1/2+δ = m > 0
Thus, all assumptions of Theorem 3.

Numerical experiment

Below we present an application of SLOPE in signal denoising. In our example X ∈ R 300×100 is an orthogonal system of trigonometric functions, i.e. Analogously we proceed with the debiased LASSO. However, in this method we use the LASSO pattern matrix defined in a following way: For LASSO we have the LASSO pattern that is a vector of signs, see Chapter 5. For S ∈ {-1, 0, 1} p , ∥S∥ 1 denotes the number of nonzero coordinates. If ∥bS∥ 1 = k ≥ 1, then we define the corresponding pattern matrix U S ∈ R p×k by

U S = diag(S) supp(S) ,
i.e. the submatrix of diag(S) obtained by keeping columns corresponding to indices in supp(S).

Then we define the reduced matrix X S by

X S = XU S .
Equivalently, we have

X S = (S i X i ) i∈supp(S)
. For a broader discussion on the pattern matrix, we encourage to see the next Chapter. In our example ε ∈ N (0, σ 2 I n ) and σ = 30. We compare the Mean Square Error and the signal denoising of the classical OLS estimation, the LASSO with the tuning parameter λ cv minimizing the cross-validated error, the debiased version of LASSO with λ = 5λ cv and the debiased version of SLOPE with the tuning vector Λ chosen with respect to the sequence proposed below Theorem 3.4.3 (λ i = 0.1(p + 1 -i)n 2/3 ). We also compare debiased SLOPE with debiased LASSO based on a single trial, as shown in Figure 3.4 and Table 3.1. The horizontal lines correspond to the true values of β. As one may observe, in the presented setting LASSO does not recover the true support, while debiased SLOPE perfectly recovers support, sign and clusters.

Appendix

Proof of Lemma 3.2.1. Since the matrix (X ′ X) -1 is nonnegative definite, it follows that the function g :

C Λ → [0, ∞) defined by g(π) := (X ′ Y -π) ′ (X ′ X) -1 (X ′ Y -π)
is convex in π. Therefore, at the point π * = (π * 1 , . . . , π * p ) ′ , where g attains its global minimum over C Λ , the gradient ∇g of g satisfies

[ ∇g(π * ) ] ′ (π -π * ) ≥ 0, for all π ∈ C Λ . This implies (π -π * ) ′ β * ≤ 0, for all π ∈ C Λ , because ∇g(π * ) = -2(X ′ X) -1 (X ′ Y -π * ) = -2β * .
In the proof of parts (a), (b) and (c), we use the fact that π * maximizes π ′ β * over π ∈ C Λ . To prove part (a), suppose that sign(β * i ) • sign(π * i ) < 0 for some i and define

π = (π * 1 , . . . , π * i-1 , -π * i , π * i+1 , . . . , π * p ) ′ .
Then we have (π

* ) ′ β * < π ′ β * , which is impossible since π ∈ C Λ .
To prove part (b), consider a permutation τ of (

π ′ β * = p i=1 |π * τ (i) ||β * i | > p i=1 |π * i ||β * i | ≥ (π * ) ′ β * , which is impossible since π ∈ C Λ . Finally, to prove part (c), suppose that k-1 i=1 |π * τ (i) | < k-1 i=1 λ i , and that |π * τ (k) | > 0.
In this case there is a sufficiently small δ > 0, such that

π = (π * 1 , . . . , π * i-2 , π * i-1 + δs i-1 , π * i -δs i , π * i+1 , . . . , π * p ) ′ ∈ C Λ . If |β * τ (k-1) | > |β * τ (k) | then π ′ β * = (π * ) ′ β * + δ(|β * τ (k-1) | -|β * τ (k) |) > (π * ) ′ β * ,
which is impossible.

Proof of Lemma 3.2.2. At first we note that for all

π ∈ C Λ r(β * , π) = 1 2 ∥Y -Xβ * ∥ 2 2 + π ′ β * = 1 2 ∥Y -Xβ * ∥ 2 2 + (π * ) ′ β * +(π -π * ) ′ β * = r(β * , π * ) + (π -π * ) ′ β * ≤ r(β * , π * ),
where the last inequality follows from the fact that (ππ * ) ′ β * ≤ 0, for all π ∈ C Λ , see the proof of 3.2.1. Therefore, max

1 n ∥Y -Xb∥ 2 2 = 1 n Y ′ Y - 2 n Y ′ Xb + b ′ b ∥ β OLS -b∥ 2 2 = 1 n 2 Y ′ XX ′ Y - 2 n Y ′ Xb + b ′ b.
Therefore, both optimization problems differ by 1 2n (Y ′ Y -1 n Y ′ XX ′ Y ), which does not depend on b, which implies their equivalence.

Chapter 4

Pattern recovery by SLOPE

Introduction

Most of the content of this chapter may be found in the preprint of Bogdan, Dupuis, Graczyk, Kołodziejek, the author of the dissertation, Tardivel and Wilczyński [START_REF] Bogdan | Pattern recovery by SLOPE[END_REF]. We decided to enrich the chapter with geometrical explanation of the SLOPE pattern and its recovery to make it more accessible for the general audience and to inform the reader about the connections between the SLOPE estimator and the convex geometry. Such connections exist also for LASSO and may be useful in its study. The geometrical approach was the one, which made us build the intuition, which led us to establish the theoretical results. In this chapter we give a deeper and more general focus on recovering the pattern of β by SLOPE. Now we abstain from restricting to the case of the orthogonal design matrix and we give novel results on the pattern recovery for a broad generality of situations. In particular, unlike in the previous chapter, below we concern a model, in which the error is incremental. We also extend the discussion on the properties of the SLOPE estimator to the high-dimensional case, in which p may be much larger than n.

History of SLOPE

SLOPE estimator was introduced by Bogdan et al. [START_REF] Bogdan | Statistical estimation and testing via the sorted l1 norm[END_REF][START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF]. In their seminal paper, SLOPE was motivated by the control of the expected rate of false discoveries (FDR control) in multiple hypotheses testing. Moreover, they proposed a choice of Λ being a Benjamini-Hochberg sequence dependent on a parameter q ∈ (0, 1), under which, in the orthogonal design, SLOPE controls FDR at a level q • p-k p , where k is the number of non-zero coordinates of β. They also proposed a fast proximal algorithm computing βSLOP E in the orthogonal design. However, the first non-trivial example of SLOPE was introduced a few years earlier by Bondell and Reich [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF] under the name OSCAR. Their article deals with the tuning vector Λ being an arithmetic sequence. The OSCAR penalty was reformulated in terms of a sorted ℓ 1 norm by Zeng and Figueiredo [START_REF] Zeng | Solving oscar regularization problems by fast approximate proximal splitting algorithms[END_REF]. Going with the flow, they independently propose the SLOPE estimator [START_REF] Zeng | Decreasing weighted sorted ℓ 1 regularization[END_REF] under the name OWL. In [START_REF] Figueiredo | Ordered weighted l1 regularized regression with strongly correlated covariates: Theoretical aspects[END_REF] they propose its representation as a gauge function, compute its dual and propose another formula for the proximal operator algorithm. Figueiredo and Nowak [START_REF] Figueiredo | Ordered weighted l1 regularized regression with strongly correlated covariates: Theoretical aspects[END_REF] prove that if the columns of X are correlated enough, then SLOPE results in their clusterization. Negrinho and Martins [START_REF] Negrinho | Orbit regularization[END_REF] connect the SLOPE estimator with the notion of a signed permutahedron C Λ , noticing that it is a unit ball in the dual of SLOPE norm. Bellec and Tsybakov [START_REF] Bellec | Bounds on the prediction error of penalized least squares estimators with convex penalty[END_REF] prove the equality X β = Y -P roj C Λ/n (Y ) in the linear regression penalized by norm. They also propose novel oracle inequalities on a prediction error, given ε ∼ N (0, σ 2 I n ) and λ j = σ log( 2p j )/n. Su and Candès [START_REF] Su | Slope is adaptive to unknown sparsity and asymptotically minimax[END_REF] prove that with Λ being the Benjamini-Hochberg sequence, SLOPE is an asymptotically minimax estimator and show more detailed results on the asymptotics of its squared error for orthogonal (X ′ X = I p ) and gaussian (X ij iid ∼ N (0, 1 n )) designs. Moreover, they point out such sizes of the support of β (ℓ 0 -sparsities), for which the risk for SLOPE is much smaller than for LASSO or SURE. Bellec, Lecué and Tsybakov [START_REF] Bellec | Slope meets Lasso: improved oracle bounds and optimality[END_REF] extend the results of Su and Candès [START_REF] Su | Slope is adaptive to unknown sparsity and asymptotically minimax[END_REF] on achieving the minimax optimal prediction rate of SLOPE. In particular, they allow the desing matrix X to be deterministic, leaving only mild assumptions on its eigenvalues. They also derive sharp oracle inequalities and prove that the minimaxity holds for an ℓ q -estimation error for any 1 ≤ q ≤ 2. Under the gaussian design, Kos and Bogdan [START_REF] Kos | On the asymptotic properties of SLOPE[END_REF] use the results of Su and Candès [START_REF] Su | Slope is adaptive to unknown sparsity and asymptotically minimax[END_REF] on the asymptotics of SLOPE in order to propose conditions on the cardinality and magnitude of nonzero coordinates of β, under which the SLOPE with Λ being a Benjamini-Hochberg sequence asymptotically achieves F DR → 0 and the power converging to 1. Hu and Lu [START_REF] Hu | SLOPE for sparse linear regression: asymptotics and optimal regularization[END_REF] put their effort to look for an optimal choice of Λ in terms of type I error and power of variable selection. They consider an asymptotic scenario, in which n p → δ > 0, k p → ρ > 0 and the vectors β, ε and Λ converge to limiting measures. Sankaran, Bach and Bhattacharrya [START_REF] Sankaran | Identifying Groups of Strongly Correlated Variables through Smoothed Ordered Weighted L 1 -norms[END_REF] prove that SLOPE belongs to the family of Lovász extensions. They also propose the smoothed version of SLOPE (called SOWL) and derive sufficient condition for its consistency. Minami [START_REF] Minami | Degrees of freedom in submodular regularization: A computational perspective of Stein's unbiased risk estimate[END_REF] studies the projections onto the dual norm ball to study the notion of degrees of freedom. SLOPE is used here as one of examples of submodular norm regularizations. Schneider and Tardivel [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] propose a geometrical approach to the SLOPE estimator. Firstly, they give full characterization of the uniqueness of βSLOP E that for a given X holds for any Y ∈ R n . They express this condition both in analytical and geometrical way. This is a generalization of an analogous result for LASSO [START_REF] Ewald | Uniformly valid confidence sets based on the Lasso[END_REF]. They also introduce the notion of SLOPE pattern and highlight its connection with faces of the unit ball in the dual J * Λ of the SLOPE norm. In their article, the definition of accessibility of the SLOPE pattern is also introduced and enriched with its geometrical full characterization. Tardivel, Servien and Concordet [START_REF] Tardivel | Simple expressions of the lasso and slope estimators in low-dimension[END_REF] propose an exact formula for SLOPE in the orthogonal case, using the Cesàro summation. For gaussian designs, random β, random ε and for n p p→∞ → δ > 0, Zhang and Bu [START_REF] Zhang | Efficient designs of slope penalty sequences in finite dimension[END_REF] use the projected gradient descent algorithm to find Λ, which reduces the Mean Squared Error (MSE) and compare obtained MSE with ones achieved by SLOPE with Benjamini-Hochberg Λ and by LASSO. To minimize MSE for arbitrary data, they propose finding Λ with help of a coordinate descent algorithm, while imposing the restriction of Λ to contain no more than k distinct values with k << p. In their recent article, Larsson et al. [START_REF] Larsson | Coordinate descent for slope[END_REF] improved solving numerically the SLOPE optimization problem, using modifications of the proximal gradient descent and the proximal coordinate descent algorithm. Bu et al. [START_REF] Bu | Algorithmic analysis and statistical estimation of SLOPE via approximate message passing[END_REF] proposed an approximate message passing (AMP) algorithm, to quickly obtain a solution, which they prove to converge in ℓ 2 to βSLOP E . They also refine the asymptotic results of the ℓ 2 -convergence of βSLOP E to the true β. Larsson, Bogdan and Wallin [START_REF] Larsson | The strong screening rule for slope[END_REF] introduced the strong screening rule for SLOPE, which generalized the analogous rule for LASSO. The safe screening rules were proposed by Bao, Gu and Huang [START_REF] Bao | Fast oscar and owl with safe screening rules[END_REF] as well as by Elvira and Herzet [START_REF] Elvira | Safe rules for the identification of zeros in the solutions of the slope problem[END_REF]. Their safety is understood as no detection any non-zero coordinate of β as zero. It is useful in situation, when some prior knowledge of β is available. Brzyski et al. [START_REF] Brzyski | Group SLOPE-adaptive selection of groups of predictors[END_REF] introduce the group SLOPE method. Lee, Sobczyk and Bogdan [START_REF] Lee | Structure learning of gaussian markov random fields with false discovery rate control[END_REF] apply SLOPE to gaussian graphical models, proposing the Neighborhood Selection SLOPE (nsSLOPE) algorithm with the FDR control of edges detection. Mazza-Anthony, Mazoure and Coates [START_REF] Mazza-Anthony | Learning Gaussian graphical models with ordered weighted ℓ 1 regularization[END_REF] independently introduce the graphical SLOPE method (under the name gOWL), which applies the SLOPE norm to estimate the precision matrix by clustering its off-diagonal entries. Recently, Riccobello et al. [START_REF] Riccobello | Graphical modelling via the sorted l1-norm[END_REF] extended the discussion on graphical SLOPE to t-Student (Tslope) data. Other applications of SLOPE include e.g. the paper of Sepehri [START_REF] Sepehri | The bayesian slope[END_REF], who proposes the bayesian approach in which SLOPE is considered as a maximum a posteriori (MAP) distribution in a bayesian regression problem. Its adaptive version was later proposed by Jiang et al. [START_REF] Jiang | Adaptive bayesian slope: Model selection with incomplete data[END_REF]. Stucky and van de Geer [START_REF] Stucky | Sharp oracle inequalities for square root regularization[END_REF] introduce the Square Root SLOPE, in which ∥Y -Xb∥ 2 2 is replaced by ∥Y -Xb∥ 2 . It is an analogue of the Square Root LASSO [START_REF] Belloni | Pivotal estimation via square-root Lasso in nonparametric regression[END_REF]. Recently, Dexheimer and Strauch [START_REF] Dexheimer | On lasso and slope drift estimators for lévy-driven ornsteinuhlenbeck processes[END_REF] found a way to apply the results of [START_REF] Bellec | Slope meets Lasso: improved oracle bounds and optimality[END_REF] and used SLOPE to improve the estimation of the drift parameter in Lévy-driven Ornstein-Uhlenbeck processes.

Our contribution

In this chapter, we give the necessary and sufficient conditions for recovering the SLOPE pattern. From a mathematical perspective, our main result is Theorem 4.3.1, in which we propose the positivity and the subdifferential condition for the pattern recovery. To ease the interpretation, we also present the dual norm condition and the SLOPE irrepresentability condition, which is a necessary and sufficient condition for pattern recovery in the noiseless case. The word "irrepresentability" is a tribute to works written a decade ago on sign recovery by LASSO [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF][START_REF] Meinshausen | High-dimensional graphs and variable selection with the Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF]. However, we believe that our mathematical perspective is novel, and paves the path for similar analyses of other penalized estimators. For the noisy case, in Theorem 4.5.1 we give the open SLOPE irrepresentability condition, which implies the consistency of the SLOPE pattern for fixed X and the difference between non-equal coordinates of |β| diverging to ∞. Later on, in Theorems 4.5.3 and 4.5.5, we give conditions for asymptotic pattern recovery, when p is fixed and n diverges to ∞. As another main results of this chapter, we provide geometrical interpretations of irrepresentability conditions as well as refined results of the strong consistency of βSLOP E and of its pattern.

Motivation

While the SLOPE ability to identify the pattern of the vector of regression coefficients β is interesting by itself, the related reduction of model dimension also brings the advantage in terms of the precision of the estimation of β. This phenomenon is illustrated in Figure 4.1, which presents the difference in precision of the LASSO and SLOPE estimators when some of the regression coefficients are equal to each other. In this example n = 100, p = 200, and the rows of the design matrix are generated as independent binary Markov chains, with P(X i1 = 1) = P(X i1 = -1) = 0.5 and P(X i(j+1) ̸ = X ij ) = 1 -P(X i(j+1) = X ij ) ≈ 0.0476. This value corresponds to the probability of the crossover event between genetic markers spaced every 5 centimorgans. To be more specific, it is close to the inverse of the Haldane mapping function [81, pp. 13-14] evaluated at 0.05, which is equal to e -0.05 sinh(0.05). Our design matrix can be viewed as an example of 100 independent haplotypes, each resulting from a single meiosis event. In this example, the correlation between columns of the design matrix decays exponentially, ρ(X i , X j ) ≈ 0.9048 |i-j| . The design matrix is then standardized so that each column has a zero mean and a unit variance, and the response variable is generated according to the linear model Y = Xβ + ε, with β 1 = . . . = β 30 = 40, β 31 = . . . = β 200 = 0 and σ = 5. In this experiment, the data matrix X and the regression model (a) q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 36 38 40 [START_REF] Candès | Near-ideal model selection by ℓ 1 minimization[END_REF]

Cluster

β estimator q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q LASSO SLOPE (b) are constructed such that the LASSO irrepresentability condition holds. The tuning parameter for LASSO is selected as the smallest value of λ for which LASSO can correctly identify the sign of β. Similarly, the tuning parameter Λ is designed such that the SLOPE irrepresentability condition holds and Λ is multiplied by the smallest constant for which SLOPE properly returns the SLOPE pattern. The selected tuning parameters for LASSO and SLOPE are represented in the left panel of Figure 4.1. Both in the case of LASSO and SLOPE, the proposed tuning parameters are close to the values minimizing the mean squared estimation error. Since in this example both LASSO and SLOPE properly estimate null components of β at 0, the right panel in Figure 4.1 illustrates only the accuracy of the estimation of the nonzero coefficients. Here, we can observe that the SLOPE ability to identify the cluster structure leads to superior estimation properties. SLOPE estimates the vector of regression coefficients β virtually without an error, while LASSO estimates are scattered over the interval between 36 and 44. In the result, the squared error of the LASSO estimator is more than 100 times larger than the squared error of SLOPE (63.4 vs 0.53).

Preliminaries on clustering properties by SLOPE

As the central notion of this chapter is the recovery of the SLOPE pattern, we recall its definition: (λ 1 , λ 2 , λ 3 , λ 4 , λ 5 ) ′ ∈ R 5+ . Then the clustered matrix and the clustered parameter are:

U M κ =        0 0 1 0 1 0 0 0 -1 0 0 0 1 0 0           8 5 2    = b.

Clustered design matrix and clustered parameter

If M = patt(β) for β ∈ R p satisfies ∥M ∥ ∞ < p,
X M = (X 5 |X 2 |X 1 -X 3 ) and Λ M =    λ 1 λ 2 λ 3 + λ 4    .

Sorted ℓ 1 norm, dual sorted ℓ 1 norm and subdifferential

As SLOPE is based on penalizing the residual sum of squared with the sorted ℓ 1 norm, we recall its definition:

J Λ (b) = p i=1 λ i |b| (i) , b ∈ R p ,
where |b| (1) ≥ . . . ≥ |b| (p) are the sorted components of b with respect to the absolute value.

Given a norm ∥•∥ on R p , we recall that the dual norm ∥•∥ * is defined by ∥b∥ * = max{v ′ b : ∥v∥ ≤ 1}, for some b ∈ R p . In particular, the dual sorted ℓ 1 norm has an explicit expression given in [START_REF] Negrinho | Orbit regularization[END_REF]:

J * Λ (b) = max |b| (1) λ 1 , 2 i=1 |b| (i) 2 i=1 λ i , . . . , p i=1 |b| (i) p i=1 λ i .
We

∂J Λ (b) = v ∈ R p : J * Λ (v) ≤ 1 and U ′ M v = Λ M . (4.2.2)
In Proposition 4.9.1 we derive a simple characterization of elements of ∂J Λ (b). The notion of SLOPE pattern is related to the subdifferential via the following result. A proof of Proposition 4.2.2 can be found in [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]. In the Appendix, we provide an independent proof, which is based on Proposition 4.2.1.

From now, to comply with Proposition 4.2.2, we assume that the tuning parameter Λ = (λ 1 , . . . , λ p ) ′ satisfies λ 1 > . . . > λ p > 0.

Characterization of SLOPE solutions

SLOPE estimator is a solution of the following optimization problem:

S X,Λ (Y ) = arg min b ∈ R p 1 2 ∥Y -Xb∥ 2 2 + J Λ (b) . (4.2.3)
We do not assume that S X,Λ (Y ) is a singleton. However, note that the cases in which the SLOPE estimator is not unique are very rare. Indeed, the family of matrices X ∈ R n×p , for which there exists a Y ∈ R n such that S X,Λ (Y ) is not a singleton, has a null Lebesgue measure on R n×p [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF].

If ker(X) = {0}, then S X,Λ (Y ) consists of one element. Recall that a convex function f attains its minimum at a point b if and only if 0 ∈ ∂f (b). Since ∂ 1 2 ∥Y -Xb∥ 2 2 = {-X ′ (Y -Xb)}, the SLOPE estimator admits the following characterization:

β ∈ S X,Λ (Y ) ⇔ X ′ (Y -X β) ∈ ∂J Λ ( β).
(4.2.4)

Characterization of pattern recovery by SLOPE

The characterization of pattern recovery by SLOPE given in Theorem 4.3.1 is one of most important results of this chapter. We recall that

P M = ( X ′ M ) + X ′ M = X M X + M
is the orthogonal projection onto col( X M ), where A + denotes the Moore-Penrose inverse of the matrix A.

Theorem 4.3.1. Let X ∈ R n×p , 0 ̸ = β ∈ R p , Y = Xβ + ε, ε ∈ R n and Λ ∈ R p+ . Let M = patt(β) ∈ M SLOPE p and k = ∥M ∥ ∞ . Denote π = X ′ ( X ′ M ) + Λ M + X ′ (I n -P M )Y . ( 4

.3.1)

There exists β ∈ S X,Λ (Y ) with patt( β) = patt(β) if and only if the two conditions below hold:

there exists s ∈ R k+ such that X ′ M Y -Λ M = X ′ M X M s, (positivity condition) π ∈ ∂J Λ (M ).
(subdifferential condition)

If the positivity and subdifferential conditions are satisfied, then β = U M s ∈ S X,Λ (Y ) and π = X ′ (Y -X β).

Remark 4.3.2.

(a) When X is deterministic and ε has a N(0, σ 2 I n ) distribution, then the pattern recovery by SLOPE is the intersection of statistically independent events:

A = ω ∈ Ω : there exists s ∈ R k+ such that X ′ M Y (ω) -Λ M = X ′ M X M s , B = {ω ∈ Ω : π(ω) ∈ ∂J Λ (M )} . Indeed, since X ′ M = X ′ M P M then X ′ M Y (ω) depends on ε A (ω) = P M ε(ω). Moreover, π(ω) depends on ε B (ω) = (I n -P M )ε(ω)
. Since P M is an orthogonal projection, then both ε A and ε B have null covariance matrices. But ε is Gaussian and hence ε A and ε B are independent. Therefore the events A and B are independent.

(b) Under the positivity condition, the subdifferential condition is equivalent to

J * Λ (π) ≤ 1. (dual norm condition) Indeed, observe that Λ M ∈ col( X ′ M ) (or equivalently, X ′ M ( X ′ M ) + Λ M = Λ M
) is necessary for the positivity condition. In view of (4.2.2), using the definition of π, we see that

U ′ M π = Λ M is equivalent to X ′ M ( X ′ M ) + Λ M = Λ M .
This follows from the fact that P M is the projection matrix onto the vector subspace col( X M ), and thus

0 ′ = [(I n -P M ) X M ] ′ = U ′ M X ′ (I n -P M ). (c)
The assertion of Theorem 4.3.1 cannot be strengthened. Indeed, when the SLOPE solution is not unique, the elements of S X,Λ (Y ) may have different SLOPE patterns.

Even if many theoretical properties on sign recovery by LASSO are known (see e.g. [START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF]), we believe that it is relevant to give a characterization of sign recovery by LASSO similar as the characterization of pattern recovery by SLOPE given in Theorem 4.3.1.

To show the similarity between proposed irrepresentability condition for SLOPE and the irrepresentability condition for LASSO [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF], we propose analogous definitions to the clustered design matrix and parameter.

1 k = (1, . . . , 1) ′ ∈ R k .
Similarly as in the proof of Theorem 4.3.1, one may prove that the necessary and sufficient conditions for the LASSO sign recovery (i.e. existence of estimator βLASSO such that sign( βLASSO ) = sign(β) = S) are the following

   there exists κ ∈ R k + such that X ′ S Y -λ S = X ′ S X S κ, (positivity condition) X ′ ( X ′ S ) + 1 k + 1 λ X ′ (I n -X S X + S )Y ∈ ∂∥ • ∥ 1 (

S). (subdifferential condition)

In the noiseless case, when ε = 0 and Y = Xβ, the subdifferential condition reduces to X ′ ( X

′ S ) + 1 k ∈ ∂∥ • ∥ 1 (S) (or equivalently, ∥X ′ ( X ′ S ) + 1 k ∥ ∞ ≤ 1 and 1 k ∈ col( X ′ S )). More- over, if we have ker(X S ) = {0}, then 1 k ∈ col( X ′ S ) and therefore ∥X ′ ( X ′ S ) + 1 k ∥ ∞ ≤ 1 is equiv- alent to ∥X ′ I X I (X ′ I X I ) -1 S I ∥ ∞ ≤ 1
, where I = supp(S), I = {1, . . . , p} \ I and X I (resp. X I ) denotes the submatrix of X obtained by keeping columns corresponding to indices in I (resp I). This latter expression is known as the irrepresentability condition [START_REF] Fuchs | On sparse representations in arbitrary redundant bases[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF].

From now on, in the definition of SLOPE (4.2.3), we consider the penalty term J Λ (b) (with a fixed Λ ∈ R p+ ) to be multiplied by a scaling parameter α > 0 and we denote the set of SLOPE solutions by S X,αΛ (Y ).

SLOPE irrepresentability condition

As illustrated by Fuchs [82, Theorem 2], Bühlmann and van de Geer [41, Theorem 7.1] and also reminded in Remark 4.3.3, the irrepresentability condition is related to sign recovery by LASSO in the noiseless case. Analogously, studying pattern recovery by SLOPE in the noiseless case allows us to introduce the SLOPE irrepresentability condition. The latter condition will be very useful in the discussed later case when ε is no longer null. Corollary 4.3.1, which provides a characterization of pattern recovery by SLOPE in the noiseless case, is a consequence of Theorem 4.3.1.

J * Λ (X ′ ( X ′ M ) + Λ M ) ≤ 1 and Λ M ∈ col( X ′ M ) (or equivalently, to X ′ ( X ′ M ) + Λ M ∈ ∂J Λ (M ))
. Moreover, under this condition, there exists α 0 > 0 such that for all α ∈ (0, α 0 ) there exists β ∈ S X,αΛ (Xβ) for which patt( β) = patt(β). Now we are ready to define the SLOPE irrepresentability condition.

Definition 4.3.2. Let M = patt(β). We define the SLOPE irrepresentability condition as the following inequality and inclusion:

J * Λ X ′ ( X ′ M ) + Λ M ≤ 1 and Λ M ∈ col( X ′ M ). (4.3.3) Remark 4.3.4. If ker( X M ) = {0}, then X ′ ( X ′ M ) + = X ′ X M ( X ′ M X M ) -1 and consequently the SLOPE irrepresentability condition reads J * Λ (X ′ X M ( X ′ M X M ) -1 Λ M ) ≤ 1.
Example 4.3.5. Let p = 2, Λ = (4, 2) ′ , β = (5, 0) ′ and β = (5, 3) ′ . Consider a design matrix

X = (X 1 |X 2 ) ∈ R n×2 satisfying X ′ X = 1 0.6 0.6 1 .
• The SLOPE irrepresentability condition does not occur when β = (5, 0

) ′ . Indeed, M = patt(β) = (1, 0) ′ , X M = X 1 (thus X ′ M X M = 1) and Λ M = λ 1 = 4. Therefore J * Λ (X ′ ( X ′ M ) + Λ M ) = J * Λ (X ′ X M ( X ′ M X M ) -1 Λ M ) = J * Λ (4X ′ X M ) = 6.4/6 > 1.
• The SLOPE irrepresentability condition occurs when β = (5, 3) ′ . Indeed, M = patt( β) = (2, 1) ′ , X M = X and Λ M = Λ. Therefore ker( X M ) = {0} and

J * Λ (X ′ ( X ′ M ) + Λ M ) = J * Λ (X ′ X(X ′ X) -1 Λ) = J * Λ (Λ) = 1 ≤ 1.
Figure 4.2 corroborates graphically that SLOPE irrepresentability condition does not hold for β, but it is satisfied for β. Note that, in this setup, the SLOPE solution is unique, since ker(X) = {0}. By β(α) we denote the unique element of S X,αΛ (Xβ). Then the SLOPE solution path is the function (0, ∞) ∋ α → β(α).

Geometrical interpretation of Irrepresentability Condition

Let 0 ̸ = β ∈ R p . By (4.2.4), for a SLOPE minimizer β ∈ S X,αΛ (Xβ) the following occurs:

1 α X ′ X(β -β) ∈ ∂J Λ ( β).
In addition, for M = patt( β), the following facts hold:

• β -β ∈ col(U M ), thus 1 α X ′ X(β -β) ∈ X ′ X col(U M ). • ∂J Λ ( β) = ∂J Λ (M ).
Therefore, the noiseless pattern recovery by SLOPE implies that the vector space X ′ Xcol(U M ) = col(X ′ X M ) intersects ∂J Λ (M ). Actually, the vector Π = X ′ ( X ′ M ) + Λ M appearing in Corollary 4.3.1 may be interpreted in a geometrical way as we propose below.

Proposition 4.4.1. Let X ∈ R n×p , 0 ̸ = M ∈ M SLOPE p and Λ ∈ R p+ . Recall that X M = XU M , Λ M = U ′ |M| ↓ Λ and Π = X ′ ( X ′ M ) + Λ M . Then: i) If Λ M / ∈ col( X ′ M ), then aff(∂J Λ (M )) ∩ col(X ′ X M ) = ∅. ii) If Λ M ∈ col( X ′ M ), then aff(∂J Λ (M )) ∩ col(X ′ X M ) = { Π}.
iii) Pattern recovery by SLOPE for ε = 0 is equivalent to

col(X ′ X M ) ∩ ∂J Λ (M ) ̸ = ∅.
In other words, the accessibility condition means that the intersection of col(X ′ X M ) ∩ aff(∂J Λ (M )) is not empty. Moreover, then it is equal to the vector Π = X ′ ( X ′ M ) + Λ M . Moreover, when the accessibility condition holds, by Proposition 4.4.1 iii) the noiseless recovery of the SLOPE pattern is equivalent to the subdifferential condition Π ∈ ∂J Λ (M ).

β 1 β 2 ∂J Λ (M ) col(X ′ X M ) Π Figure 4.3: Intersection Π of col(X ′ X M ) = col((1, 2/3) ′ ) and aff(∂J Λ (M )) for X ′ X = 1 2/3 2/3 1 , Λ = (4, 1) ′ and patt(β) = M = (1, 0) ′ . Since Π = (4, 8/3) ′ / ∈ ∂J Λ (M ), then
in the noiseless case, by Proposition 4.4.1, SLOPE does not recover M . However, as Π exists, the pattern M is accessible, as in the latter examples.

Proof. i) Recall that, according to Lemma 4.9.2, aff(∂J

Λ (M )) = {v ∈ R p : U ′ M v = Λ M }. If aff(∂J Λ (M )) ∩ col(X ′ X M ) ̸ = ∅, then there exists z ∈ R k , where k = ∥M ∥ ∞ , such that X ′ X M z ∈ aff(∂J Λ (M )). Consequently, Λ M = U ′ M X ′ X M z = X ′ M X ′ M z, thus Λ M ∈ col( X ′ M ). ii) If Λ M ∈ col( X ′ M ), then Π ∈ aff(∂J Λ (M )). Indeed, since X ′ M ( X ′ M ) + is the projection onto col( X ′ M ), we have U ′ M Π = X ′ M ( X ′ M ) + Λ M = Λ M .
Moreover, since col(( X ′ M ) + ) = col( X M ), we deduce that Π ∈ col(X ′ X M ). To prove that Π is the unique point in the intersection, we will show that col(X

′ X M )∩col(U M ) ⊥ = {0}. Indeed, if v ∈ col(X ′ X M ) ∩ col(U M ) ⊥ , then there exists such z ∈ R k that v = X ′ X M z = X ′ XU M z and (U M z) ′ v = 0. Therefore, (U M z) ′ X ′ XU M z = 0, which implies that XU M z = X M z = 0 and thus v = 0. Finally, if Π ∈ aff(∂J Λ (M )) ∩ col(X ′ X M ), then Π -Π ∈ col(X ′ X M ) and U ′ M (Π -Π) = 0, which implies that Π = Π.
iii) According to Corollary 4.3.1, pattern recovery by SLOPE in the noiseless case is equivalent to Π ∈ ∂J Λ (M ) which is equivalent, by i) and ii), to col(X ′ X M ) ∩ ∂J Λ (M ) ̸ = ∅.

Proposition 4.4.1 may be interpreted with a notion of the accessibility condition for SLOPE, which was introduced in [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF].

Definition 4.4.1 (Accessible pattern). [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] Let X ∈ R n×p , λ > 0 and pen be a polyhedral gauge. We say that the pattern of β ∈ R p is accessible with respect to X and λpen, if there exist y ∈ R n and β ∈ S X,λpen (y) such that β pen ∼ β.

The accessibility of a pattern can be characterized in a geometric and an analytic way. In the low-dimensional setting, when the design matrix X is of a full column rank, another geometrical interpretation of βSLOPE might be used. As a corollary of Theorem 3.2.2, one may deduce that the difference between βSLOP E and βOLS is equal to

(X ′ X) -1 π * ∈ (X ′ X) -1 C Λ ,
where C Λ is the unit ball in J * Λ and π * is defined as in Lemma 3.2.1. On Figure 4.7 we present graphically this relation between βSLOP E and βOLS .

The illustration of the SLOPE dual norm unit ball C Λ in R p for p = 3 may be found in the recently published article of Schneider and Tardivel [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]. We recall it to present that with growing number of explanatory variables p the complexity of C Λ rises drastically.

(-1,-2) (-2,-1) (-1,0) (-1,1) (0,1) (1,1)
(1,0)

(1,-1) (0,-1) (-1,-1) (0,0)

′ X) -1 Xε ∈ (X ′ X) -1 C Λ for p = 2, X ′ X = 1 2/3 2/3 1
and Λ = (4, 1) ′ .

Asymptotics of pattern recovery and pattern consistency

In this section we consider two asymptotic scenarios and establish conditions on tuning parameters for which the pattern of β is recovered. In Section 4.5.1 we consider the case where gaps between distinct absolute values of β diverge and in Section 4.5.2 the case where the sample size n diverges. The proofs rely on Theorem 4.3.1. We show that the positivity and subdifferential conditions are satisfied under our settings. It turns out that for the positivity condition the tuning parameter cannot be too large, while for the subdifferential condition it cannot be too small. In this way we consider a tuning parameter of the form αΛ, where Λ ∈ R p+ is fixed and α varies. We determine the assumptions for the sequence (α) for which both positivity and subdifferential conditions hold true, i.e. for which the pattern is recovered.

X is a fixed matrix

The subdifferential condition, given in Theorem 4.3.1, says that a vector π defined in (4.3.1) belongs to ∂J αΛ (M ), where α is a scaling parameter. This condition is equivalent to requiring

β 1 β 2 ∂J Λ (M ) col(X ′ X M ) Π Figure 4.9: Intersection Π of col(X ′ X M ) = col((1, 1/4) ′ ) and aff(∂J Λ (M )) for X ′ X = 1 1/4 1/4 1 , Λ = (4, 1) ′ and patt(β) = M = (1, 0) ′ . Since Π = (4, 1) ′ ∈ F M = ∂J Λ (M ),
but it lies on a boundary of F M . Therefore the open SLOPE irrepresentability condition does not hold, even though the irrepresentability condition is satisfied.

that a vector π/α belongs to ∂J Λ (M ). We denote the vector π/α by

π α = X ′ ( X ′ M ) + Λ M + 1 α X ′ (I n -P M )Y = X ′ ( X ′ M ) + Λ M + 1 α X ′ (I n -P M )ε, (4.5.1)
where in the latter equality we use the fact that (I n -P M ) is an orthogonal projection onto col( X M ) ⊥ and therefore (I

X ′ ( X ′ M ) + Λ M ∈ ri (∂J Λ (M )) .

We illustrate the difference between the SLOPE irrepresentability condition and its open version on

β (r) = U M s (r) with s (r) ∈ R k+ and k = ∥M ∥ ∞ ,
whose strength is increasing in the following sense:

∆ r = min 1≤i<k s (r) i -s (r) i+1 r→∞ -→ ∞, with the convention s (r) k+1 = 0
and let Y (r) = Xβ (r) + ε, where ε is a vector in R n .

(a) Sharpness of the upper bound: Let α > 0. If ε is random, then the upper bound (4.5.2) is asymptotically reached:

lim r→∞ P ∃ β ∈ S X,αΛ (Y (r) ) such that patt( β) = M = P (J * Λ (π α ) ≤ 1) , 0 if Λ M / ∈ col( X ′ M ). (b) Pattern consistency: If α r → ∞, α r /∆ r → 0 as r → ∞ and X ′ ( X ′ M ) + Λ M ∈ ri(∂J Λ (M )),
then for any ε ∈ R n we have

∃ r 0 > 0 ∀ r ≥ r 0 ∃ β ∈ S X,αrΛ (Y (r) ) such that patt( β) = M . Remark 4.5.2. (a) The open SLOPE irrepresentability condition X ′ ( X ′ M ) + Λ M ∈ ri(∂J Λ (M )
) is equivalent to the following computationally verifiable conditions:

   J * Λ (X ′ ( X ′ M ) + Λ M ) ≤ 1 and Λ M ∈ col( X ′ M ), i ∈ {1, . . . , p} : i j=1 |X ′ ( X ′ M ) + Λ M | (j) = i j=1 λ j = ∥M ∥ ∞ .
This equivalence follows from Proposition 4.9.1.

(b) Let us assume that the distribution of ε and -ε is equal. Because the unit ball of the dual sorted ℓ 1 norm is convex, when J * Λ (X ′ ( X ′ M ) + Λ M ) > 1 then, independently on α > 0, the probability of pattern recovery is smaller than 1/2, namely

P ∃ β ∈ S X,αΛ (Y ) such that patt( β) = M ≤ 1/2.
For LASSO, a similar inequality on the probability of sign recovery is given in [START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF].

(c) In Section 4.7, we illustrate that, under the open irrepresentability condition, one may select α > 0 to fix the asymptotic probability of pattern recovery at a level arbitrarily close to 1 (a similar result for LASSO is given in [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF]).

X is random, p is fixed, n tends to infinity

In this section we discuss asymptotic properties of the SLOPE estimator in the low-dimensional regression model in which p is fixed and the sample size n tends to infinity.

For each n ≥ p we consider a linear regression problem

Y n = X n β + ε n , (4.5.3)
where X n ∈ R n×p is a random design matrix. We will use the following assumptions:

A. ε n = (ε 1 , . . . , ε n ) ′ ,
(n) ij | n i=1 (X (n) ij ) 2 P -→ 0.
C. (X n ) n and (ε n ) n are independent.

We will consider a sequence of tuning parameters (Λ n ) n defined by

Λ n = α n Λ,
where Λ ∈ R p+ is fixed and (α n ) n is a sequence of positive numbers.

Let βSLOPE

n be an element of the set S Xn,Λn (Y n ) of SLOPE minimizers. Under Assumption B1, for large n with high probability, the set S Xn,Λn (Y n ) consists of one element. Indeed, we have

P (ker(X n ) = {0}) = P X ′ n X n is positive definite n→∞ -→ 1
and ker(X n ) = {0} ensures existence of the unique SLOPE minimizer. In a natural setting, the strong consistency of βSLOPE n can be characterized in terms of behaviour of the tuning parameter. We use such approach in the Theorem 3.4.1 and Theorem 4.6.2. At this point we note that if (4.6.8) holds almost surely, then the condition α n /n → 0 ensures that βSLOPE -→ patt(β). However, if ∥patt(β)∥ < p, then the situation is more complex as we shall show below. The first of our asymptotic results concerns the consistency of the pattern recovery by the SLOPE estimator. We note that Assumption B2 is not necessary for the SLOPE pattern recovery. This assumption was introduced to ensure the existence of a Gaussian vector in the Theorem 4.5.3 (a).

lim n→∞ P patt( βSLOPE n ) = patt(β) = P (J * Λ (V ) ≤ 1) , where V ∼ N(CU M (U ′ M CU M ) -1 Λ M , σ 2 [C -CU M (U ′ M CU M ) -1 U ′ M C]). (b) Let CU M (U ′ M CU M ) -1 Λ M ∈ ri(∂J Λ (M )). (4.5.5)
The pattern of SLOPE estimator is consistent, i.e.

patt( βSLOPE

n ) P -→ patt(β), if and only if lim n→∞ α n n = 0 and lim n→∞ α n √ n = ∞.
(c) The condition

J * Λ CU M (U ′ M CU M ) -1 Λ M ≤ 1 (4.5.6)
is necessary for pattern consistency of SLOPE estimator.

The random vector V belongs to the smallest affine space containing ∂J Λ (b), i.e. aff(∂J Λ (b)) = {v ∈ R p : U ′ M v = Λ M }, see Lemma 4.9.2. Condition (4.5.5) is the open SLOPE irrepresentability condition in the n → ∞ regime. The above result should be compared with [START_REF] Zhao | On model selection consistency of Lasso[END_REF]Theorem 1], where the same conditions on the LASSO tuning parameter ensure consistency of sign recovery by LASSO estimator. Below we make one step further and consider the strong consistency of SLOPE pattern recovery by βSLOPE n . Although this was not the main focus of Zhao and Yu, it can be deduced from [START_REF] Zhao | On model selection consistency of Lasso[END_REF]Theorem 1] that if for c ∈ (0, 1) the LASSO tuning parameter λ n satisfies λ n /n → 0 and λ n /n -→ sign(β). Even though the patterns are discrete objects, as the underlying probability space is uncountable, the convergence in probability does not imply the almost sure convergence. Below we show that if

α n /n → 0 and α n / √ n → ∞, then patt( βSLOPE n
) is not strongly consistent and one actually has to impose a slightly stronger condition (4.5.7). For the purpose of the a.s. convergence, we strengthen the assumption on design matrices: B'. Assume that the rows of X n are independent and that each row of X n has the same law as Ξ, where Ξ is a random vector whose components are linearly independent almost surely and that E[Ξ 2 i ] < ∞ for i = 1, . . . , p. Remark 4.5.4. Under B', by the strong law of large numbers, we have n -1 X ′ n X n a.s.

-→ C, where -→ patt(β). (4.5.8) Remark 4.5.6. Assume that (4.5.5) is satisfied and set α n = c √ n log log n for c > 0. Then (4.5.7) is not satisfied and the probability that the correct SLOPE pattern is not recovered is greater than zero. In Section 4.6.1 we present more refined results on the strong consistency of the SLOPE pattern. The log log n correction in (4.5.7) comes from the law of iterated logarithm.

C = (C ij ) ij with C ij = E[Ξ i Ξ j ].

Strong consistency of SLOPE and its pattern 4.6.1 Refined results on strong consistency of the SLOPE pattern

In this section we aim to give assumptions on the design matrix that are weaker than condition B', but they ensure the almost sure convergence of the pattern of βSLOPE B". A sequence of design matrices X 1 , X 2 , . . . satisfies the condition

1 n X ′ n X n a.s. -→ C, (4.6.2)
where C is a deterministic positive definite symmetric p × p matrix.

With

X n = X (n) ij ij , lim n→∞ (log n) ρ √ n sup i,j X (n) ij
= 0 a.s. for all ρ > 0 (4.6.3) and there exist nonnegative random variables

(c i ) i , constants d > 2/r and m 0 ∈ N such that for n > m ≥ m 0 , sup j   m i=1 X (n) ij -X (m) ij 2 + n i=m+1 X (n) ij 2   ≤   n i=m+1 c i   d a.s., (4.6.4)   n i=m 0 c i   d = O(n) a.s. (4.6.5) C. (X n ) n and (ε n ) n are independent.
We note that conditions (4.6.3) and (4.6.4) are trivially satisfied in the i.i.d. rows setting of Remark 4.5.4 or Assumption B'. The main ingredient of the proof of the strong pattern consistency is the law of iterated logarithm (4.9.9) which holds trivially under B'. Below we establish the same result under more general B". The technical assumption (4.6.4) is a kind of weak continuity assumption on the rows of X n as it says that the ℓ 2 distance between j th rows of X n and X m should not be too large.

Lemma 4.6.1. Assume A', B" and C. Then

lim sup n→∞ ∥X ′ n ε n ∥ ∞ √ n log log n < ∞ a.s. (4.6.6)
Proof. In view of (4.6.8) we have for j = 1, . . . , p,

n -1 A (j) n := n -1 n i=1 X (n) ij 2 = n -1 X ′ n X n jj a.s.
-→ C jj > 0. (4.6.7)

We apply the general law of iterated logarithm for weights forming a triangular array from [START_REF] Lai | A law of the iterated logarithm for double arrays of independent random variables with applications to regression and time series models[END_REF].

The result follows directly from [117, Theorem 1], which we recall in Theorem 2.3.18. Defining a

(j) ni := X (n)
ij for i = 1, . . . , n, j = 1, . . . , p, n ≥ 1 and 0 otherwise, we have

(X ′ n ε n ) j = ∞ i=-∞ a (j)
ni ε i and therefore we fall within the framework of (2.3.5). Then, (4.6.

P   lim sup n→∞ |(X ′ n ε n ) j | 2A (j) n log log A (j) n ≤ σ (X n ) n   = 1 a.s.
Averaging over (X n ) n and using (4.6.7) again, we obtain the assertion. -→ patt(β).

Comments:

(a) Under reasonable assumptions (see e.g. [117, Theorem 1 (iii)]), one can show that lim sup

n→∞ ∥X ′ n ε n ∥ ∞ √ n log log n > 0 a.s. Since α -1 n X ′ n ε n a.s.
-→ 0 is necessary for the a.s. pattern recovery, we can show that the condition α n / √ n log log n → ∞ cannot be weakened. Thus, the gap between the convergence in probability and the a.s. convergence is integral to the problem and in general cannot be reduced.

(b) One can relax assumption B" by imposing stronger conditions on the error ε n . For example, if ε n is Gaussian, then one can use results from [START_REF] Stadtmüller | A note on the law of iterated logarithm for weighted sums of random variables[END_REF]. We note that [START_REF] Stadtmüller | A note on the law of iterated logarithm for weighted sums of random variables[END_REF] offers a very similar result as [START_REF] Lai | A law of the iterated logarithm for double arrays of independent random variables with applications to regression and time series models[END_REF], but their assumptions are not quite comparable, see [167, Section 3 i)] for detailed discussion.

(c) For Gaussian errors, one can consider a more general setting where one does not assume any relation between ε n and ε n+1 , i.e. the error need not be incremental. For orthogonal design such approach was taken in [START_REF] Skalski | Pattern recovery and signal denoising by slope when the design matrix is orthogonal[END_REF]. It is proved there that one obtains the a.s. SLOPE pattern consistency with the second limit condition of Theorem 4.6.1 replaced by lim n→∞ α n / √ n log n = ∞. This result can be generalized to non-orthogonal designs.

Strong consistency of the SLOPE estimator

Lemma 4.6.2. Let ε n = (ε 1 , . . . , ε n ) ′ with ε i i.i.d

., centered and having finite variance.

Suppose that

1 n X ′ n X n a.s.
-→ C > 0. (4.6.8)

and that (ε n ) n and (X n ) n are independent. Then n -1 X ′ n ε n a.s.
-→ 0.

Proof. Denote by P( • | (X n ) n ) the regular conditional probability. By [52, Th. 1.1] applied to the sequence (n -1 X ′ n ε n ) j on the probability space (Ω, F, P(

• | (X n ) n )), we obtain P lim n→∞ n -1 (X ′ n ε n ) j = 0 | (X n ) n = 1, j = 1, . . . , p, a.s.
Thus, applying the expectation to both sides above we obtain the assertion.

Y n = X n β + ε n if and only if for π n = X ′ n (Y n -X n βSLOPE n ) we have J * Λ (π n ) ≤ 1 (4.6.10)
and (4.6.11) where

U ′ Mn π n = Λ n ,
M n = patt( βSLOPE n ) and Λ n = U ′ |M n|↓ Λ n . By the definition of π n we have βSLOPE n = (X ′ n X n ) -1 X ′ n Y n -(X ′ n X n ) -1 π n = βOLS n - 1 n X ′ n X n -1 1 n π n .
Since in our setting βOLS n is strongly consistent, βSLOPE n a.s.

-→ β if and only if

(n -1 X ′ n X n ) -1 n -1 π n a.s.
-→ 0. In view of (4.6.8), we have (n

-1 X ′ n X n ) -1 n -1 π n a.s.
-→ 0 if and only if n -1 π n a.s.

-→ 0. Assume n -1 λ (n) 1 → 0. By (4.6.10) we have ∥π n ∥ ∞ ≤ λ (n) 1 , which gives π n n ∞ ≤ λ (n) 1 n → 0.
Therefore, (4.6.9) implies that βSLOPE n a.s.

-→ β. Now assume that β ̸ = 0 and βSLOPE is strongly consistent, i.e. n -1 π n a.s.

-→ 0. Then, (4.6.11) gives

p∥π n ∥ ∞ ≥ ∥U ′ Mn π n ∥ ∞ = ∥ Λ n ∥ ∞ ≥ λ (n) 1 (4.6.12)
provided M n ̸ = 0. Applying (4.6.10) for βSLOPE n = 0, we note that M n (ω) = 0 if and only if

J * n -1 Λn n -1 X n (ω) ′ Y n (ω) ≤ 1.
In view of Lemma 4.6.2, it can be easily verified that n

-1 X ′ n Y n a.s. -→ Cβ. Since 1 n π n ∞ ≥ 1 n π n ∞ 1 {Mn=0} = 1 n X ′ n Y n ∞ 1 {Mn=0} ,
we see that for β ̸ = 0, we have M n ̸ = 0 for large n almost surely. Thus, for β ̸ = 0 we eventually obtain for large n λ

(n) 1 n ≤ p π n n ∞ a.s.

Simulation study

Our simulations aim at illustrating Theorems 4.5.1 and 4.5.3 and at showing that the results provided in these theorems are somehow unified. We consider the linear regression model Y = Xβ + ε, where X ∈ R n×p and ε ∈ R n has i.i.d. N(0, 1) entries. Up to a constant, we choose components of Λ = (λ 1 , . . . , λ p ) ′ as expected values of ordered standard Gaussian statistics. Let V (1) ≥ . . . ≥ V (p) be ordered statistics of i.i.d. N(0, 1) random variables. An approximation of E[V (i) ] for some i ∈ {1, . . . , p}, denoted E(i, p), is given below (see [START_REF] Harter | Expected values of normal order statistics[END_REF] and references therein)

E(i, p) = -Φ -1 i -0.375 p + 1 -0.750 ,
where Φ is the cumulative distribution function of a N(0, 1) random variable. We set

λ i = E(i, p) + E(p -1, p) -2E(p, p) for i = 1, . . . , p. Note that since E(1, p) > . . . > E(p, p), then Λ = (λ 1 , . . . , λ p ) ′ ∈ R p+ .
For the design matrix X and the vector of regression coefficients β we consider two cases:

• X is orthogonal, and the components of β are all equal with a magnitude that tends to infinity.

• X is asymptotically orthogonal, the sample size diverges to infinity and the components of β are equal to 1.

Sharp upper bound when X is orthogonal

In Figure 4.10 we have p = 100, X ∈ R n×p is orthogonal (i.e X ′ X = I 100 ) and β 1 = . . . = β p = c > 0. To compute the value α 0.95 of the scaling parameter for which the upper bound equals 0.95, we note that π α is a Gaussian vector of N X ′ ( X

′ M ) + Λ M , α -2 X ′ (I -X M X + M )X distribution. Moreover, since M = patt(β) = (1, . . . , 1) ′ , we have X ′ ( X ′ M ) + Λ M = 1 p p i=1 λ i , . . . , 1 p p i=1
λ i and (4.7.1)

X ′ (I n -X M X + M )X =        1 -1/p -1/p . . . -1/p -1/p . . . . . . . . . . . . . . . . . . -1/p -1/p . . . -1/p 1 -1/p       
.

Since the distribution of π α is given and the open SLOPE irrepresentability condition occurs, for arbitrary q ∈ (0, 1) one can choose α q for which P(J * Λ (π αq ) ≤ 1) = q. In the following graph, q = 0.95 and α 0.95 ≈ 1.391.

Limiting probability when X is asymptotically orthogonal

In

Discussion

This chapter makes an important step in understanding the clustering properties of SLOPE and we have shown that the irrepresentability condition provides theoretical guarantees for SLOPE pattern recovery. However, this by no means closes the topic of the SLOPE pattern recovery. Similarly to the irrepresentability condition for LASSO, SLOPE irrepresentability condition is rather stringent and imposes a strict restriction on the number of nonzero clusters in β. On the other hand, in [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF] it is shown that a much weaker condition for LASSO is required to separate q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 0 10 20 30 40 Cluster β estimator q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q LASSO SLOPE the estimators of the null components of β from the estimators of nonzero regression coefficients. This condition, called accessibility (also called identifiability), requires that the vector β has a minimal ℓ 1 norm among all vectors γ such that Xβ = Xγ. Thus, when the accessibility condition is satisfied, one can recover the sign of β by thresholding LASSO estimates. Empirical results from [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF] suggest that this weaker condition is also sufficient for the sign recovery by the adaptive LASSO [START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF]. In Chapter 5 it is shown that a similar result holds for SLOPE. In this case, the accessibility condition is satisfied if the vector β has the smallest sorted ℓ 1 norm among all vectors γ such that Xβ = Xγ. In Chapter ?? it is shown that when the accessibility condition is satisfied then SLOPE properly ranks the estimators of regression coefficients and the SLOPE pattern can be recovered by shrinking similar estimates towards the cluster centers. Figure 4.12 illustrates this phenomenon and shows that the accessibility condition for SLOPE can be much less restrictive than the accessibility condition for LASSO. In this example, the matrix X and the vector Y are generated as in the example illustrated in Figure 4.1 and the only difference is that now first k = 100 = n regression coefficients are all equal to 40. In this situation, the accessibility condition for LASSO is not satisfied and LASSO can not properly separate the null and nonzero regression coefficients. Also, despite the selection of the tuning parameter so as to minimize the squared estimation error, the precision of LASSO estimates is very poor. As far as SLOPE is concerned, the irrepresentability condition is not satisfied, but the accessibility condition holds. Thus, while SLOPE can not properly identify the pattern, it estimates β with such a good precision that the difference between the estimated and the true pattern is hardly visible on the graph. These nice ranking and estimation properties of SLOPE bring a promise for efficient pattern recovery by appropriate thresholded and adaptive SLOPE versions.

4.9 Appendix -Proofs

J Λ (b) = λ 1 |b| (1) + . . . + λ p |b| (p) = Λ ′ U |M| ↓ s = Λ ′ s = s 1 Λ 1 + . . . + s k Λ k .
Moreover, with p l = |{i :

|m i | ≥ k + 1 -l}|, we have Λ l = λ p l-1 +1 + . . . + λ p l , l = 1, . . . , k.
Proof of Proposition 4.2.1.

First, we prove the inclusion ∂J

Λ (b) ⊂ v ∈ R p : J * Λ (v) ≤ 1 and U ′ M v = Λ . Let v ∈ ∂J Λ (b). Since J * Λ (v) ≤ 1 (see (4.2.1
)) then, by definition of the dual sorted ℓ 1 norm, for all j ∈ {1, 2, . . . , p} we have j i=1 |v| (i) ≤ j i=1 λ i . It remains to prove that U ′ M v = Λ. For all l ∈ {1, . . . , k} we have the following inequality

l i=1 [U ′ M v] i = i : |m i |≥k+1-l sign(m i )v i ≤ i : |m i |≥k+1-l |v i | ≤ p l i=1 |v| (i) ≤ p l i=1 λ i = l i=1 Λ i . (4.9.1) Note that b ′ v = (U M s) ′ v = k i=1 s i [U ′ M v] i = k-1 l=1 (s l -s l+1 ) l i=1 [U ′ M v] i + s k k i=1 [U ′ M v] i ≤ k-1 l=1 (s l -s l+1 ) l i=1 Λ i + s k k i=1 Λ i = k l=1 s l Λ l = J Λ (b). Moreover, since v ∈ ∂J Λ (b), we have b ′ v = J Λ (b) (see (4.2.1)). Therefore, l i=1 [U ′ M v] i = l i=1 Λ i for l = 1, . . . , k
and thus the inequalities given in (4.9.1) are the equalities. Thus

[U ′ M v] l = Λ l for l = 1, . . . , k and hence that U ′ M v = Λ.

Now we prove the other inclusion, ∂J

Λ (b) ⊃ v ∈ R p : J * Λ (v) ≤ 1 and U ′ M v = Λ M . Assume that v ∈ R p satisfies J * Λ (v) ≤ 1 and U ′ M v = Λ. To prove that v ∈ ∂J Λ (b), it remains to establish that b ′ v = J Λ (b) (see (4.2.1)). Since b = U M s, we have b ′ v = (U M s) ′ v = s ′ U ′ M v = s ′ Λ = J Λ (b).

Proof of Proposition 4.2.2

Lemma 4.9.1.

Let Λ ∈ R p+ and b ∈ R p . If Λ ∈ ∂J Λ (b), then b 1 ≥ . . . ≥ b p ≥ 0.
Proof. Let us assume that b i < 0 for some i ∈ {1, . . . , p}.

For π = (λ 1 , . . . , λ i-1 , -λ i , λ i+1 , . . . , λ p ) we have J * Λ ( π) ≤ 1 and one may deduce that

Λ ′ b < π′ b ≤ max{π ′ b : J * Λ (π) ≤ 1} = J Λ (b).
Consequently Λ / ∈ ∂J Λ (b), which leads to a contradiction. Now, let us assume that b i < b j for some 1 ≤ i < j ≤ p. We define π as the following:

πk =        λ k if k ̸ = i, k ̸ = j, λ j if k = i, λ i if k = j, k = 1, . . . , p.
Note that J * Λ ( π) ≤ 1. Since λ i > λ j , by the rearrangement inequality we have λ i b i + λ j b j < λ j b i + λ i b j . Thus, one may deduce the following inequality

Λ ′ b < π′ b ≤ max{π ′ b : π ∈ R p J * Λ (π) ≤ 1} = J Λ (b).
= |M | ↓ namely M 1 ≥ M 2 ≥ . . . ≥ M p ≥ 0. In this case, M ′ Λ = J Λ (M ) and hence Λ = (λ 1 , . . . , λ p ) ′ ∈ ∂J Λ (M ). Since ∂J Λ (M ) = ∂J Λ ( M ), it follows from Lemma 4.9.1 that M 1 ≥ . . . ≥ M p ≥ 0, because Λ ∈ ∂J Λ ( M ). To prove that M = M , first let us establish that M p = M p = 0 or M p = M p = 1. If M p = 0 and M p = 1, then we set π = (λ 1 , . . . , λ p-1 , 0) ′ , where J * Λ ( π) ≤ 1. Because J Λ (M ) = Λ ′ M = π′ M and J Λ ( M ) = Λ ′ M > π′ M ,
we have π ∈ ∂J Λ (M ) and π / ∈ ∂J Λ ( M ) which provides a contradiction. We proceed analogously for M p = 1 and M p = 0. To achieve proving that M = M , let us establish that

m i = m i+1 and m i = m i+1 or m i > m i+1 and m i > m i+1 . If m i = m i+1 and m i > m i+1 then we define π satisfying J * Λ ( π) ≤ 1, as follows πk =        λ k if k ̸ = i, k ̸ = i + 1, λ i+1 if k = i, λ i if k = i + 1, k = 1, . . . , p. Since λ i m i + λ i+1 m i+1 = λ i+1 m i + λ i m i+1 and λ i m i + λ i+1 m i+1 > λ i+1 m i + λ i m i+1 then J Λ (M ) = Λ ′ M = π′ M and J Λ ( M ) = Λ ′ M > π′ M .
Consequently π ∈ ∂J Λ (M ) and π / ∈ ∂J Λ ( M ), which provides a contradiction. We proceed analogously for m i > m i+1 and m i = m i+1 . Finally, if M ̸ = |M | ↓ , then we may pick an orthogonal transformation ψ as defined above for which

ψ(M ) = |M | ↓ . Since ∂J Λ (M ) = ∂J Λ ( M ) implies that ∂J Λ (ψ(M )) = ∂J Λ (ψ( M )),
(a) If b i ̸ = 0, then sign(x i ) = sign(b i ), (b) If |b i | > |b j | then |x i | ≥ |x j |, (
By Proposition 4.2.2, X ′ (Y -X β) ∈ ∂J Λ ( β) = ∂J Λ (M ). Multiplying this inclusion by U ′ M , due to (4.2.2), we get X ′ M (Y -X β) = Λ M and so X ′ M Y -Λ M = X ′ M X β = X ′ M X M s.
(4.9.

3)

The positivity condition is proven. We apply ( X ′ M ) + from the left to (4.9.3) and use the fact that

P M = ( X ′ M ) + X ′ M is the pro- jection onto col( X M ). Since X β ∈ col( X M ), we have P M X β = X β. Thus, P M Y -( X ′ M ) + Λ M = X β.
The above equality gives the subdifferential condition:

∂J Λ (M ) ∋ X ′ (Y -X β) = X ′ (Y -( P M Y -( X ′ M ) + Λ M )) (4.9.4) = X ′ ( X ′ M ) + Λ M + X ′ (I n -P M )Y = π.
Sufficiency. Assume that the positivity condition and the subdifferential conditions hold true.

Then, by the positivity condition, one may pick s ∈ R k+ for which

Λ M = X ′ M Y -X ′ M X M s. (4.9.5)
Let us show that U M s ∈ S X,Λ (Y ). By definition of U M , we have patt(U M s) = M , thus ∂J Λ (U M s) = ∂J Λ (M ). Moreover, using (4.9.4) and (4.9.5) one may deduce that

∂J Λ (U M s) ∋ π = X ′ (Y -P M Y + ( X ′ M ) + Λ M ) = X ′ (Y -P M Y + ( X ′ M ) + ( X M Y -X ′ M X M s)) = X ′ (Y -XU M s).
Consequently, U M s ∈ S X,Λ (Y ).

Proof of Corollary 4.3.1

Proof of Corollary 4.3.1. If SLOPE recovers the pattern of β in the noiseless case, then, by Theorem 4.3.1, the subdifferential condition reads as:

X ′ ( X ′ M ) + Λ M ∈ ∂J Λ (M ). Conversely, if X ′ ( X ′ M ) + Λ M ∈ ∂J Λ (M )
k = ∥M ∥ ∞ , we have X ′ M Y -α Λ M = X ′ M X M s -α Λ M . Therefore for α > 0 small enough, X ′ M Y -α Λ M ∈ X ′ M X M R
(∂J Λ (b)) ⊂ {v ∈ R p : U ′ M v = Λ M }.
Moreover, according to Theorem 4 in [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] we have

dim(aff(∂J Λ (b))) = ∥M ∥ ∞ = dim({v ∈ R p : U ′ M v = Λ M }),
which achieves the proof.

Proof of Theorem 4.5.1. (i) Sharpness of the upper bound. According to Theorem 4.3.1, the pattern recovery by SLOPE is equivalent to have simultaneously the positivity condition and the subdifferential condition satisfied. The upper bound (4.5.2) coincides with the probability of the subdifferential condition. Thus to prove that this upper bound is sharp, it remains to show that the probability of the positivity condition tends to 1 when r tends to ∞. Clearly the upper bound is reached when

Λ M / ∈ col( X ′ M ), thus we may assume that Λ M ∈ col( X ′ M ). Recall that β (r) = U M s (r) for s (r) ∈ R k+ and thus X ′ M Y (r) = X ′ M X M s (r) + X ′ M ε. As X ′ M ( X ′ M ) + = X ′ M X M ( X ′ M X M ) + is the projection on col( X ′ M ), we obtain X ′ M Y (r) -α r Λ M = X ′ M X M s (r) -α r Λ M + X ′ M ε = X ′ M X M s (r) -α r X ′ M X M ( X ′ M X M ) + Λ M + X ′ M X M ( X ′ M X M ) + X ′ M ε = X ′ M X M ∆ r 1 ∆ r s (r) - α r ∆ r ( X ′ M X M ) + Λ M + 1 ∆ r ( X ′ M X M ) + X ′ M ε .
Note that by the assumption on ∆ r :

• the vector s (r) /∆ r ∈ R k+ is component-wise larger than or equal to (k, . . . , 1),

• lim r→∞ α r /∆ r = 0 and lim r→∞ 1/∆ r = 0.

Consequently, for r large enough we have

X ′ M Y (r) -α r Λ M ∈ X ′ M X M R k+ .
Since this fact is true for any realization of ε, we get

lim r→∞ P X ′ M Y (r) -α r Λ M ∈ X ′ M X M R k+ = 1.
(ii) Pattern consistency. In the proof of the previous part, we see that positivity condition occurs when r is sufficiently large. Thus it remains to prove that subdifferential condition occurs as r → ∞ when X ′ ( X

′ M ) + Λ M ∈ ri(∂J Λ (M )). First we observe that X ′ ( X ′ M ) + Λ M + 1 α r X ′ (I n -P M )ε r→∞ -→ X ′ ( X ′ M ) + Λ M . ( 4.9.6)
Note by Lemma 4.9.2 that X ′ ( X

′ M ) + Λ M + α -1 r X ′ (I n -P M )ε ∈ aff(∂J Λ (M )). Indeed, since Λ M ∈ col( X ′ M ), we have U ′ M X ′ ( X ′ M ) + Λ M = Λ M + 1 α r U ′ M X ′ (I n -P M )ε(ω) =0 = Λ M .
The second term above is zero due to the fact that (I n -P M ) is an orthogonal projection onto col( X M ) ⊥ . When X ′ ( X ′ M ) + Λ M ∈ ri(∂J Λ (M )), due to (4.9.6), one may deduce that for sufficiently large r we have

X ′ ( X ′ M ) + Λ M + 1 α r X ′ (I n -P M )ε ∈ ∂J Λ (M ).
Consequently, when r is sufficiently large, both the positivity and the subdifferential conditions occur which, by Theorem 4.3.1, concludes the proof.

Proofs from Section 4.5.2

In this part we give proofs of Theorem 4.5.3 and Theorem 4.5.5. They are preceded by a series of simple lemmas. For reader's convenience we recall the setting of Section 4.5.2.

A.

ε n = (ε 1 , . . . , ε n ) ′ , where (ε i ) i are i.i.d. centered with finite variance σ 2 . B1. n -1 X ′ n X n P -→ C > 0. B2. max i=1,...,n |X (n) ij | n i=1 (X (n) ij ) 2 P -→ 0, where X n = (X (n) ij ) ij , for each j = 1, . . . , p.
B'. Rows of X n are i.i.d. distributed as Ξ, where Ξ is a random vector whose components are linearly independent a.s. and such that E[Ξ

2 i ] < ∞ for i = 1, . . . , p. C. (X n ) n and (ε n ) n are independent.
We consider a sequence of tuning parameters (Λ n ) n defined by Λ n = α n Λ, where Λ ∈ R p+ is fixed and (α n ) n is a sequence of positive numbers.

To ease the notation, we write the clustered matrices and clustered parameters without the subscript indicating the pattern M , i.e. Λ := U ′ |M |↓ Λ, Λ n := α n Λ and X n := X n U M . Lemma 4.9.3.

(a) Under A, B1, B2 and C,

1 √ n X ′ n ε n d -→ V ∼ N(0, σ 2 C). (4.9.7) (b) Under A, B1 and C, 1 n X ′ n ε n P -→ 0. (4.9.8) (c) Under A, B' and C, 0 < lim sup n→∞ ∥X ′ n ε n ∥ ∞ √ n log log n < ∞ a.s. ( 4
.9.9)

Proof of (4.9.7). It is enough to show that for any Borel subset A ⊂ R p one has

P 1 √ n X ′ n ε n ∈ A | (X n ) n P -→ P (V ∈ A) . (4.9.10)
Since both sides above are bounded, the convergence in probability implies convergence in L 1 and therefore establishes (4.9.7). To show (4.9.10), we will prove that for any subsequence (n k ) k , there exists a sub-subsequence (n k l ) l for which, as l → ∞,

P 1 √ n k l X ′ n k l ε n k l ∈ A | (X n ) n a.s.
-→ P (V ∈ A) .

(4.9.11)

Let P X denote the regular conditional probability P(• | (X n ) n ) on (Ω, F). By assumptions B1 and B2, from sequences (n k ) k one can choose a subsequence (n k l ) l for which

1 n k l X ′ n k l X n k l a.s. -→ C > 0 and max i=1,...,n k l |X (n k l ) ij | n k l i=1 (X (n k l ) ij ) 2 a.s -→ 0, j = 1, . . . , p.
We have

Var X 1 √ n k l X ′ n k l ε n k l = 1 n k l E X ′ n k l ε n k l ε ′ n k l X n k l | (X n ) n = 1 n k l X ′ n k l E ε n k l ε ′ n k l X n k l = σ 2 n k l X ′ n k l X n k l a.s. -→ σ 2 C > 0,
and one can apply multivariate Lindeberg-Feller CLT on the space (Ω, F, P X ) to prove (4.9.11).

Alternatively, the same result follows from Theorem 2.3.16, which concerns more general Central Limit Theorem for linearly negative quadrant dependent variables with weights forming a triangular array (in particular assumption B2 coincides with (2.3.3)). For our application, the assumption of nonnegative weights in Theorem 2.3.16 is not essential. For (ii) we observe that previous derivations imply that Var X (n -1 X ′ n ε n ) P -→ 0. We deduce that P X (n -1 X ′ n ε n ) P -→ 0 and hence (ii) follows upon averaging over (X n ) n . Eq. (4.9.9) is the law of iterated logarithm for an i.i.d. sequence (Ξ i ε i ) i . Proof. If M = 0, then the positivity condition is trivially satisfied. Thus, we consider M ̸ = 0. (i) Since X ′ n X n is invertible for large n with high probability, the positivity condition is equivalent to

s n := ( X ′ n X n ) -1 [ X ′ n Y n -Λ n ] ∈ R k+ . Let s 0 ∈ R k+ be defined through β = U M s 0 , where k = ∥M ∥ ∞ . We will show that if α n /n → 0, then s n P -→ s 0 . Since R k+
is an open set, this will imply that for large n with high probability, the positivity condition is satisfied. First we rewrite s n as

s n = ( X ′ n X n ) -1 X ′ n Y n -α n ( X ′ n X n ) -1 Λ. Since β = U M s 0 , we conclude X n β = X n U M s 0 = X n s 0 , so the linear regression model takes the form Y n = X n s 0 + ε n . Thus, ( X ′ n X n ) -1 X ′ n Y n is the OLS estimator of s 0 .
By assumption B and Lemma 4.9.3, we deduce that

( X ′ n X n ) -1 X ′ n Y n = s 0 + (n -1 X ′ n X n ) -1 U M 1 n X ′ n ε n P -→ s 0 + [(U ′ M CU M ) -1 U M ]0 = s 0 .
To complete the proof, we note that

α n ( X ′ n X n ) -1 Λ = α n n n( X ′ n X n ) -1 Λ P -→ 0 (U ′ M CU M ) -1 Λ = 0.
(ii) If one assumes B' instead of B1, then n -1 X ′ n X n a.s.

-→ C and by (4.9.9), n -1 X ′ n ε n a.s.

-→ 0. The result follows along the same lines as (i).

For M ̸ = 0 we denote π (1) (1) n + π (2) n , which simplifies in the M = 0 case to

n = X ′ n ( X ′ n ) + Λ n , π (2) n = X ′ n (I n -P n )Y n , π n = π
π n = π (2) n = X ′ n Y n . Recall that the subdifferential condition is equivalent to J * Λn (π n ) ≤ 1 and Λ n ∈ col( X ′ M
) and the latter is satisfied in our setting. Since αJ Λ = J αΛ , the subdifferential condition is satisfied if and only if

1 ≥ J * Λ α -1 n π n = J * Λ α -1 n π (1) n + √ n α n n -1/2 π (2) n .
In view of results shown below, α -1 n π

(1)

n converges almost surely, while n -1/2 π (2)
n converges in distribution to a Gaussian vector. Thus, the pattern recovery properties of SLOPE estimator strongly depend on the behavior of the sequence (α n / √ n) n .

Lemma 4.9.5. (a) (i) Assume A, B1 and C. If M ̸ = 0, then

1 α n π (1) n P -→ CU M (U ′ M CU M ) -1 Λ.
Proof of Theorem 4.5.5. By Lemma 4.9.4, the positivity condition is satisfied for large n almost surely. By Lemma 4.9.5 (i) and (iii), we have

a n := 1 α n π n a.s. -→ CU M (U ′ M CU M ) -1 Λ =: a 0 .
It is easy to see that U ′ M a n = Λ. By the condition a 0 ∈ ri(J Λ (M )) it follows that a n ∈ J Λ (M ) almost surely for sufficiently large n. Therefore π n ∈ J Λn (M ) for large n almost surely and thus the subdifferential condition is also satisfied.

Chapter 5

Geometry of Pattern Recovery by Penalized and Thresholded Estimators

Introduction

The content of this chapter is based on the preprint of Graczyk, Schneider, the author of the dissertation and Tardivel [START_REF] Graczyk | Pattern recovery in penalized and thresholded estimation and its geometry[END_REF]. As in previous chapters, we consider the linear regression model

Y = Xβ + ε,
where X ∈ R n×p is a design matrix, β ∈ R p is an unknown vector of regression coefficients and ε ∈ R n is a random noise term. We assume that the distribution of ε is symmetric, continuous and its density is strictly positive. Many penalized estimators of β have been broadly studied in literature, e.g., LASSO [START_REF] Chen | Basis pursuit[END_REF][START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF], SLOPE [START_REF] Zeng | Decreasing weighted sorted ℓ 1 regularization[END_REF][START_REF] Bogdan | SLOPE -adaptive variable selection via convex optimization[END_REF][START_REF] Negrinho | Orbit regularization[END_REF], OSCAR [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF], fused LASSO [START_REF] Tibshirani | Sparsity and smoothness via the fused lasso[END_REF], fused adaptive LASSO [START_REF] Rinaldo | Properties and refinements of the fused lasso[END_REF], clustered LASSO [START_REF] She | Sparse regression with exact clustering[END_REF], PACS [START_REF] Sharma | Consistent group identification and variable selection in regression with correlated predictors[END_REF] and generalized LASSO [START_REF] Tibshirani | Degrees of freedom in lasso problems[END_REF]. When the loss function is the residual sum of squares, these estimators minimize, with respect to b ∈ R p , a function:

f (b) = 1 2 ∥Y -Xb∥ 2 2 + λpen(b),
where λ > 0 is the tuning parameter and the penalty term pen is a real-valued polyhedral gauge, i.e. a nonnegative convex function, homogeneous, vanishing at 0 and whose unit ball is a polyhedron. Note that in this definition we do not require boundedness of a unit ball. The literature related to penalized least squares estimators is vast and many of these estimators have interesting and relevant structures as illustrated, e.g., in [START_REF] Vaiter | Model selection with low complexity priors[END_REF]. For instance, LASSO is sparse, i.e. some components of this estimator are null. The fused LASSO is a sparse and piecewise constant estimator [START_REF] Tibshirani | Sparsity and smoothness via the fused lasso[END_REF], the supremum norm promotes a cluster of components maximal in absolute value [START_REF] Jégou | Anti-sparse coding for approximate nearest neighbor search[END_REF], and SLOPE and OSCAR estimators promote clusters made of those components of these estimators, which are equal in absolute value [START_REF] Bondell | Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR[END_REF][START_REF] Figueiredo | Ordered weighted l1 regularized regression with strongly correlated covariates: Theoretical aspects[END_REF][START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF][START_REF] Bogdan | Pattern recovery by slope[END_REF][START_REF] Skalski | Pattern recovery and signal denoising by slope when the design matrix is orthogonal[END_REF].

Pattern recovery by penalized least squares estimators

We define a polyhedral gauge pen as a nonnegative convex function, homogeneous, vanishing at 0 and whose unit ball is a polyhedron. Every polyhedral gauge can be written as the maximum 75 of a finite number of linear functions [START_REF] Rockafellar | Convex Analysis[END_REF][START_REF] Mousavi | Solution uniqueness of convex piecewise affine functions based optimization with applications to constrained ℓ 1 minimization[END_REF] ∀x ∈ R p , pen(x) = max{0, u ′ 1 x, . . . , u ′ l x}, for some u 1 , . . . , u l ∈ R p . Note that a polyhedral gauge with a bounded and symmetric unit ball {x ∈ R p : pen(x) ≤ 1} is a polyhedral norm.

Given X ∈ R n×p , Y ∈ R n and λ > 0, the set S X,λpen (Y ) of solutions of a penalized least squares optimization problem is defined as follows:

S X,λpen (Y ) := arg min b ∈ R p 1 2 ∥Y -Xb∥ 2 2 + λpen(b). (5.1.1)
It is important to know that the set S X,λpen (Y ) is not empty:

Theorem 5.1.1. Let X ∈ R n×p , Y ∈ R n , λ > 0 and pen(x) = max{0, u ′ 1 x, . . . , u ′ l x}, where u 1 , . . . , u l ∈ R p . Then the function f (b) = 1 2 ∥Y -Xb∥ 2 2 + pen(b)
has at least one minimizer.

The proof is given in the Appendix. Note that, potentially, the set S X,λpen (Y ) might be not a singleton, i.e. the penalized least squares estimator might be not unique. Below we recall the definition of the pattern equivalence class, which is one of the most important notions in this chapter.

Definition 2.2.2 (Equality of patterns)

. Let pen : R p → R be a polyhedral gauge. We say that x ∈ R p and z ∈ R p have the same pattern with respect to pen, denoted x pen ∼ z, when

∂pen(x) = ∂pen(z),
where ∂pen is the subdifferential of pen. The set of all vectors having the same subdifferential as x, denoted C x , is called the pattern equivalence class.

In Theorem 5.2.2 we prove that pattern equivalence classes, illustrated in Section 2.2, are given by normal cones of B * , where B * is the polar set of the unit ball of pen.

For the ℓ 1 norm penalization, two vectors x, z ∈ R p have the same pattern if and only if sign(x) = sign(z). More generally, two vectors having the same pattern with respect to a polyhedral gauge penalty have a specific structure as illustrated on many examples in Section 5.3. Given X and Y , we aim at recovering the pattern of β. For LASSO this means the recovery of sign(β). In Theorem 5.4.4 we give a necessary condition for pattern recovery by penalized least squares estimators, called noiseless recovery condition. Later, in Section 5.5, we introduce penalized estimators relaxing this condition. Beforehand, we recall some of the already known conditions for pattern recovery.

Conditions for pattern recovery -examples

LASSO. Below we assume the uniqueness of the LASSO estimator. Then we note βLASSO as a unique element of S X,λ∥.∥ 1 (Y ). As mentioned above, LASSO estimation is a sparse method that nullifies some of the components with positive probability, entailing that the estimator also performs variable selection. Instigated by this sparsity property, an abundant literature has arisen to deal with the recovery of the location of the non-null components of β, or, more specifically, the recovery of the sign vector of β [START_REF] Fuchs | Recovery of exact sparse representations in the presence of bounded noise[END_REF][START_REF] Meinshausen | High-dimensional graphs and variable selection with the Lasso[END_REF][START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF][START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF]]. An evident necessary condition for sign recovery by LASSO is for sign(β) to be accessible by LASSO, i.e. for a fixed λ > 0, there has to exist such Y ∈ R n that sign( βLASSO (Y )) = sign(β). Otherwise, the sign recovery is impossible. A geometrical characterization of accessible sign vectors is given in [START_REF] Sepehri | The accessible lasso models[END_REF][START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]. However, the accessibility of sign(β) by LASSO does not mean that the probability of sign recovery by LASSO is close to 1 even if the non-null components of β are extremely large. Actually, for sign recovery with a probability larger than 1/2 a stronger condition is needed, called the irrepresentability condition [START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF], which is satisfied when

∥X ′ I X I (X ′ I X I ) -1 sign(β I )∥ ∞ ≤ 1,
where

I := {i ∈ [p] : β i ̸ = 0} and I := {i ∈ [p] : β i = 0}.
SLOPE. The notions of accessibility condition and irrepresentability condition for SLOPE have been recently introduced respectively in [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] and in Chapter 4. In particular, in Chapter 4, similarly as for LASSO, when the SLOPE irrepresentability condition does not occur, the probability of pattern recovery is smaller than 1/2.

Generalized LASSO. By substituting the ℓ 1 norm with a polyhedral gauge pen = ∥D.∥ 1 for a fixed matrix D of a linear map on R p , one constructs an estimator β ∈ S X,λ∥D.∥ 1 (Y ),

where D β has some null components. It is a reason why the generalized LASSO is frequently used for structure recovery. One should be aware that the structure induced by generalized LASSO depends on the matrix D.

β i ̸ = β i+1 } [104]
. Actually, articles [START_REF] Qian | On stepwise pattern recovery of the fused lasso[END_REF][START_REF] Owrang | Consistent change point detection for piecewise constant signals with normalized fused lasso[END_REF] provide theoretical properties for the jump set recovery under the irrepresentability condition.

The noiseless recovery condition and the irrepresentability condition can be relaxed using thresholded estimators as explained below.

Pattern recovery by a thresholded estimator

Theorem 5.5.1 generalizes results known for LASSO to a wide class of penalized estimators. Specifically, we prove that thresholding penalized least squares estimators allows the recovery of the pattern of β with large probability under a weaker condition than before. We recall the definition of thresholded estimator below.

Definition 2.3.23.

Let pen be a polyhedral gauge, X ∈ R n×p , Y ∈ R n and λ > 0. Given β ∈ S X,λpen (Y ), we say that β is a thresholded estimator of β if ∂pen( β) ⊆ ∂pen( β).

One of the examples of a thresholded penalized estimator is the thresholded LASSO 5.5.1. Given a threshold τ ≥ 0, the thresholded LASSO βLASSO,τ is defined as follows

βLASSO i ,τ = βLASSO i if | βLASSO i | > τ, 0 otherwise.
Note that for every τ ≥ 0 we have ∂∥.∥ 1 ( βLASSO ) ⊆ ∂∥.∥ 1 ( βLASSO,τ ) and thus βLASSO,τ is a thresholded estimator of βLASSO in the sense of Definition 2.3.23. The thresholded LASSO does not have the same statistical properties as LASSO, cf. [START_REF] Meinshausen | Lasso-type recovery of sparse representations for highdimensional data[END_REF][START_REF] Weinstein | A power analysis for knockoffs with the lasso coefficient-difference statistic[END_REF]. Concerning sign recovery by thresholded LASSO, the accessibility condition is necessary and sufficient, which was proven by Tardivel and Bogdan [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF]. Moreover, they also proved that, contrarily to LASSO, thresholded LASSO can recover the sign of β with a large probability under the accessibility condition, even if the irrepresentability condition is not satisfied, as soon as non-null components of β are sufficiently large. This nice property for sign recovery by thresholded LASSO remains true for thresholded basis pursuit [START_REF] Saligrama | Thresholded basis pursuit: Lp algorithm for order-wise optimal support recovery for sparse and approximately sparse signals from noisy random measurements[END_REF][START_REF] Descloux | Model selection with lasso-zero: adding straw to the haystack to better find needles[END_REF][START_REF] Descloux | Robust lasso-zero for sparse corruption and model selection with missing covariates[END_REF].

Geometry of pattern equivalence

Let F be a face of a polytope P . We propose the following relation between normal cones N P (x) and N F (x) for x ∈ ri(F ), that seems unknown.

Proposition 5.2.1. Let x ∈ ri(F ) ⊂ P . Then lin(N P (x)) = N F (x).
Proof. A polytope P can be represented as {v ∈ R p : ⟨s j , v⟩ ≤ r j , j = 1, . . . , m}, cf. [101, p. 138]. Then one may describe a face F ⊂ P as

F = P ∩ {v ∈ R p : ⟨-s j , v⟩ ≤ -r j , j ∈ A P (x)},
where A P (x) := {j = 1, . . . , m : ⟨s j , v⟩ = r j } is the set of active constraints for x. Without loss of generality one may rearrange the sequence {s j } m j=1 such that A P (x) = {1, . . . , a}, where a ≤ m. Then F is equal to

F = {v ∈ R p : ⟨ s j , v⟩ ≤ r j , j = 1, . . . , m + a},
where

s j = s j , j ≤ m, -s j-m , j > m and r j = r j , 1 ≤ m, -r j , j > m.
The corresponding set of active constraints A F (x) is equal to

A F (x) = A P (x) ∪ {m + 1, . . . , m + a}.
By Proposition 2.3.2 we obtain

N F (x) = cone{ s j : j ∈ A F (x)} = cone{s j , -s j : j ∈ A P (x)} = lin{s j : j ∈ A P (x)} = lin(N P (x)).
We will also use the following property of normal cones.

Proposition 5.2.2. Let x ∈ ri(F ) ⊂ P . Then the normal cone N P (x) has the property

lin(N P (x)) = ----→ aff(F ) ⊥ .
Proof. (⊂) We start by noticing that for any d ∈ lin(N P (x)) = N F (x) and for any s ∈ F we have ⟨d, s -x⟩ = 0. Indeed, since s = x -(xs) and x ∈ ri(F ), then there exists such δ > 0 that x + δ(x -s) ∈ F . Therefore,

0 = ⟨d, x -x⟩ = δ δ + 1 ⟨d, x -(x -(x -s))⟩ + 1 δ + 1 ⟨d, x -(x + δ(x -s))⟩,
(x)) ⊂ ----→ aff(F ) ⊥ . (⊃) Let v ∈ ----→ aff(F ) ⊥ Lemma 2.3.2 = lin{w -x : w ∈ F }.
Then ⟨v, x -w⟩ = 0 for every w ∈ F . It implies that v ∈ N F (x), which by Proposition 5.2.1 equals lin(N P (x)). Remark 5.2.1. In addition to the property of normal cones from Proposition 2.3.2, we deduce from Propositions 5.2.2 and 2.3.2 that lin ({s j : j Proof. Let w ∈ C x and s ∈ ri(F x ). Since s ∈ F x = F w then, for all z ∈ B * , we have

∈ A P (x)}) = ----→ aff(F ) ⊥ .

Pattern equivalence classes and normal cones

w ′ (z -s) = w ′ z ≤pen(w) -w ′ s =pen(w) ≤ pen(w) -pen(w) = 0 Consequently, w ∈ N B * (s) = N B * (F x ).
Theorem 5.2.2. The pattern cone C x is the relative interior of the normal cone of the face F x of the dual unit ball

B * C x = ri(N B * (F x )). (5.2.1)
Proof. If ∂pen(x) = F , we denote C x = C F . R p can be partitioned both into pairwise disjoint pattern sets and into pairwise disjoint relative interiors of corresponding normal cones:

F C F = R p = F ri(N (F )).
Thus it is sufficient to prove the inclusion C F ⊂ ri(N (F )), or equivalently

C x ⊂ ri(N B * (F x )) for x ∈ R p .
(

Let w ∈ C x . We want to show that w ∈ ri(N B * (F x )), i.e. there exists δ > 0 such that any point z ∈ B(w, δ) ∩ aff(N B * (F x )) belongs to C x . According to Lemma 5.8.9, for δ > 0 small enough we obtain

F z ⊆ F w , which equals F x , since w ∈ C x . Moreover, if F z ̸ = F x , then one may pick u ∈ F z ⊂ F x and v ∈ F x \ F z . Observe that u -v ∈ ----→ aff(F x ). Then, by Lemma 2.3.1, we have x ∈ C x ⊂ N B * (F x ), which by Lemma 5.8.1 belongs to ----→ aff(F x ) ⊥ . Therefore ⟨x, u -v⟩ = 0.
We also have z ∈ aff(N B * (F x )) = lin(N B * (F x )), which by Lemma 5.8.1 belongs to ----→ aff(F x ) ⊥ . Therefore ⟨z, u -v⟩ = 0, too. Consequently, v ′ z = u ′ z = pen(z) and thus v ∈ F z , which leads to a contradiction. Therefore F z = F x , which means that z ∈ C x .

Model subspace recovery

More generally, for a wide class of penalty terms including polyhedral gauges, Vaiter et al. [START_REF] Vaiter | Model selection with low complexity priors[END_REF] showed that the irrepresentability condition is sufficient for the model subspace recovery by penalized least squares estimators. The notion of model subspace is related to the notion of pattern. Specifically, the model subspace of x ∈ R p is a vector subspace of R p perpendicular to ∂pen(x). For the ℓ 1 norm two vectors x, z ∈ R p have the same model subspace when supp(x) = supp(z). In the particular case of LASSO, Theorem 6 in [START_REF] Vaiter | Model selection with low complexity priors[END_REF] shows that ∥X ′ I X I (X ′ I X I ) -1 sign(β I )∥ ∞ < 1 is a sufficient condition for model subspace recovery, i.e. the recovery of supp(β). Whereas correct, this statement is not optimal. Indeed, when ∥X ′ I X I (X ′ I X I ) -1 sign(β I )∥ ∞ < 1, it is well known that LASSO actually can recover sign(β) and a fortiori supp(β) [START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF]. Whereas we do not retain the notion of model subspace, in Theorem 5.2.3 we prove that the model subspace coincides with the linear span of a pattern equivalence class.

Definition 5.2.1 (Model subspace). ([182]) The model subspace S x of x is the orthogonal complement of

----→ aff(F x ):

S x := ----→ aff(F x ) ⊥ ,
Our objective in this section is to prove that S x = lin(C x ), i.e. the linear space generated by a pattern equivalence class coincides with the model subspace from [START_REF] Vaiter | Model selection with low complexity priors[END_REF].

Theorem 5.2.3. S x = lin(C x ).

Proof.

S x = ----→ aff(F x ) ⊥ P rop. 5.2.2 = lin(N (F x )) = lin(ri(N (F x ))) T hm. 5.2.2 = lin(C x ).
Remark 5.2.4. The equality S x = lin(C x ) does not hold for general penalizers. Indeed, when pen is a strictly convex function, then S x = R p , but lin(C x ) does not need to be equal to R p .

For example, pen(x) = ∥x∥ 2 2 gives lin

(C 0 ) = {0} ̸ = R p .
( * , 1) ( * , -1)

(-1, * ) (1, * ) (1, 1) (1, -1) (-1, 1) (-1, -1)
( * , * ) are equal if and only if patt(x) = patt(z). For instance, if x = (3.1, - Furthermore, the composition of a polyhedral gauge with a linear map is still a polyhedral gauge.

(0, 0) (1, 0) (-1, 0) (0, 1) (0, -1) (1, 1) (-1, -1) (-1, 1) (1, -1) (2, 1) (1, 2) (-2, 1) (-1, 2) (-2, -1) (-1, -2) (2, -1) (1, -2)
For example, for generalized LASSO, the penalty term is the polyhedral gauge x ∈ R p → ∥Dx∥ 1 where D ∈ R m×p . Note that, when ker(D) ̸ = 0, the function x ∈ R p → ∥Dx∥ 1 is not a norm but only a semi-norm. Below we present two examples of generalized LASSO.

• Total variation:

Let p ≥ 2 and let D tv ∈ R (p-1)×p be the first order difference matrix defined as follows

D tv =        -1 1 0 . . . 0 0 -1 1 . . . . . . . . . . . . . . . . . . 0 0 . . . 0 -1 1        .
• ℓ 1 trend filtering:

Let p ≥ 3 and let D tf ∈ R (p-2)×p be the second order difference matrix defined as follows

D tf =        -1 2 -1 0 . . . 0 0 -1 2 -1 . . . . . . . . . . . . . . . . . . . . . 0 0 . . . 0 -1 2 -1       
.

The total variation and the ℓ 1 trend filtering [START_REF] Kim | ℓ 1 trend filtering[END_REF] are examples of the generalized LASSO with the penalty term ∥D tv .∥ 1 and ∥D tf .∥ 1 , respectively.

Pattern for the total variation ∥D tv .∥ 1 :

Let p ≥ 2. The vector jump(x) ∈ {↗, →, ↘} p-1 is defined as follows Pattern for the trend filtering ∥D tf .∥ 1 :

∀i ∈ [p -1], jump(x) i :=        ↗ if x i+1 > x i → if x i+1 = x i ↘ if x i+1 < x i Subdifferentials ∂∥D tv .∥ 1 (x) = ∂∥D tv .∥ 1 (z)
Let p ≥ 3. The vector knot(x) ∈ {l, cx, cv} p-2 is defined as

follows ∀i ∈ [2 : p -1], knot(x) i :=        cx if x i < (x i+1 -x i-1 )/2, l if x i = (x i+1 -x i-1 )/2, cv if x i > (x i+1 -x i-1 )/2.
Consider the piecewise linear curve

L x := p-1 i=1 [(i, x i ), (i + 1, x i+1 )]. Note that knot(x) i is equal
to l (resp. cx or cv) when, in the neighborhood of i, the curve L x is linear (resp. convex or concave). Subdifferentials ∂∥D tf .∥ 1 (x) = ∂∥D tf .∥ 1 (z) are equal if and only if knot(x) = knot(z).

Pattern recovery in penalized estimation

Accessibility condition

Below we recall the notion of accessible patterns in the following definition, which generalizes the notion of accessible sign vectors [START_REF] Sepehri | The accessible lasso models[END_REF][START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] and accessible SLOPE patterns [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] to a broad class of estimators penalized with polyhedral gauges. Definition 5.4.1 (Accessible pattern). Let X ∈ R n×p , λ > 0 and pen be a polyhedral gauge. We say that the pattern of β ∈ R p is accessible with respect to X and λpen, if there exist y ∈ R n and β ∈ S X,λpen (y) such that β pen ∼ β.

The accessibility of a pattern can be characterized in a geometric and an analytic way. Based on Proposition 4.4.2, one may see that the notion of accessibility does not depend on the tuning parameter λ.

Noiseless recovery condition

The solution path for a penalized estimator is defined as the curve 0 < λ → β(λ) where β(λ)

is the unique element of S X,λpen (Y ) for fixed Y ∈ R n and X ∈ R n×p . The solution paths for the generalized LASSO or OSCAR and the Clustered LASSO are studied in [START_REF] Tibshirani | Degrees of freedom in lasso problems[END_REF] or [START_REF] Takahashi | Efficient path algorithms for clustered lasso and oscar[END_REF], respectively. Based on this notion, below we define the noiseless recovery condition. Note that the following definition does not require the uniqueness of an estimator. Then the LASSO solution path 0 < λ → βLASSO (λ) is the following curve:

Irrepresentability Condition for polyhedral gauges

The following theorem is one of the main results of this chapter. Its proof is based on Theorem 5.2.2.

Theorem 5.4.3. Let pen be a polyhedral norm. Let ε = 0 and β ∈ R p , β ̸ = 0. Then the existence of β ∈ S X,λpen (Xβ) recovering the pattern of β is equivalent to Moreover, we obtain 1 λ X ′ (Xβ -X β) = X ′ Xz ∈ F β . Therefore β ∈ S X,λpen (Xβ), which ends the proof.

X ′ Xlin(C β ) ∩ F β ̸ = ∅.

LASSO solution path

Below we provide a geometrical characterization of the noiseless recovery condition. Neither the above definition nor the geometrical characterization provide an analytic expression for checking the noiseless recovery condition, but for some of the penalized estimators analogous formulas have been already given. For example, when pen = ∥.∥ 1 , the noiseless recovery condition is equivalent to ∥X ′ (X ′ I ) + sign(β I )∥ ∞ ≤ 1 and sign(β I ) ∈ row(X I ), (5.4.1) where I = {i ∈ [p] : β i ̸ = 0} and X I is the matrix whose columns are (X j ) j∈I . Note that under the assumption that ker(X I ) = {0} we obtain sign(β I ) ∈ row(X I ) and the expression (5.4.1) coincides with the irrepresentability condition: ∥X ′ I X I (X ′ I X I ) -1 sign(β I )∥ ∞ ≤ 1 where X I is a matrix whose columns are (X j ) j / ∈I [START_REF] Bühlmann | Statistics for High-Dimensional Data: Methods, Theory and Applications[END_REF][START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF][START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF][START_REF] Zhao | On model selection consistency of Lasso[END_REF]. Thus, the well known irrepresentability condition for LASSO can be thought of as an analytical shortcut for checking the noiseless recovery condition, see Figure 5.6. Indeed, in the above example, we have ∥X ′ I X I (X ′ I X I ) -1 sign(β I )∥ ∞ = 30/29 > 1 and based on Figure 5.6, one may observe that the noiseless recovery condition does not hold for β. For SLOPE with M = patt(β), the noiseless recovery condition is equivalent to

J * Λ (X ′ ( X ′ M ) + Λ M ) ≤ 1 and Λ M ∈ row( X M )
, where J Λ is the sorted ℓ 1 norm, X M is the clustered matrix and Λ M is the clustered parameter, cf. Definition 2.2.5. In Appendix we also provide an analytic characterization of noiseless pattern recovery when the penalty term is the supremum norm. Below we show that (a) The noiseless recovery condition is necessary to recover the pattern with a probability larger than 1/2, see Theorem 5.

P ∃λ > 0 ∃ β ∈ S X,λpen (Y ) such that β pen ∼ β ≤ 1/2.
Corollary 5.4.1. If the noiseless recovery condition does not hold for the LASSO (for example, when

∥X ′ I X I (X ′ I X I ) -1 sign(β I )∥ ∞ > 1)
, the following holds

P ∃λ > 0 ∃ β ∈ S X,λ∥.∥ 1 (Y ) such that sign( β) = sign(β) ≤ 1/2.
The above result extends the Theorem 2 from [START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF], which shows that P(sign( βLASSO (λ)) = sign(β)) ≤ 1/2 for fixed λ > 0.

If β satisfies the noiseless recovery condition with respect to X and pen, then β is accessible with respect to X and pen, by taking y = Xβ in the definition of accessibility. In the following section, we show that thresholded penalized least-squares estimators recover the pattern of β under the accessibility condition.

Pattern recovery by thresholded estimators

In practice, some additional information about β may be priorly known, e.g. its sparsity. Therefore it is quite natural to threshold small components of βLASSO and to consider the thresholded LASSO estimator βLASSO,τ for some threshold τ ≥ 0: Definition 5.5.1 (Thresholded LASSO). The thresholded LASSO [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF] is defined in a following way [START_REF] Bühlmann | Statistics for High-Dimensional Data: Methods, Theory and Applications[END_REF]Sec. 2.9]:

βLASSO i ,τ = βLASSO i if | βLASSO i | > τ, 0 otherwise.
(5.5.1)

Moreover, if the threshold is appropriately selected, the estimator allows to recover sign(β) under weaker conditions than LASSO itself [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF]. We aim at generalizing this property to the class of penalized estimators with polyhedral penalty. Before introducing the notion of a thresholded estimator, recall that for any threshold τ ≥ 0, the inclusion ∂∥.∥ 1 ( βLASSO ) ⊆ ∂∥.∥ 1 ( βLASSO,τ ) occurs. This last inclusion is the keystone concept to introduce the notion of a thresholded estimator as defined in Definition 2.3.23. The notion of accessibility introduced for penalized estimators in Section 5.4 also covers the thresholded estimators as can be seen below. On the other hand, if the accessibility condition is satisfied, then both penalized and thresholded penalized estimator can recover the pattern of β with different choices of y. However, in practice, instead of choosing an appropriate y to recover the pattern of β, the response of a linear regression model is being used to infer this pattern. In this direction, by Theorem 5.4.4, if y = Xβ + ε, then the recovery of the pattern of β with probability larger than 1/2 requires the noiseless recovery condition, which is stronger than the accessibility condition. This result remains true for any symmetric and continuous noise. In Theorem 5.5.1, we relax the stringent noiseless recovery condition by considering a thresholded estimator. Before stating this theorem, we introduce the following class of thresholded estimators.

Definition 5.5.2 (τ -thresholded penalized estimator). Let pen be a real-valued polyhedral gauge,

X ∈ R n×p , Y ∈ R n and λ ≥ 0. Given β ∈ S X,λpen (Y ), we say that β τ is a τ -thresholded estimator of β if (a) ∂pen( β) ⊆ ∂pen( β τ ), (a) ∥ β -β τ ∥ ∞ ≤ τ , (a) dim(∂pen(b)) ≤ dim(∂pen( β τ )) for all b with ∥ β -b∥ ∞ ≤ τ .
The thresholded LASSO is, in fact, an example of a τ -thresholded estimator with the threshold τ . Another example of a τ -thresholded estimator when the penalty term is the supremum norm, is given in Algorithm 1. Theorem 5.5.1 shows that a thresholded estimator recovers the pattern of β under the accessibility condition and the assumption that the signal is large enough, as formalized in the following theorem. Similar results in which non-null components are large enough (i.e., r ≥ r 0 in Theorem 5.5.1) are given in [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF] and [START_REF] Descloux | Robust lasso-zero for sparse corruption and model selection with missing covariates[END_REF]. In particular, Theorem 5.5.1 corroborates Theorem 1 in [START_REF] Tardivel | On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising[END_REF], which proves that the thresholded LASSO estimator recovers the sign of β once the accessibility condition holds and non-null components of β are large enough. Similarly as thresholded LASSO, while pen = ∥.∥ ∞ , a τ -estimator can be explicitly computed by Algorithm 1.

Algorithm 1 Thresholded penalized estimator when the penalty term is the ℓ ∞ norm:

Require: estimate: β, threshold τ ≥ 0. if ∥ β∥ ∞ ≤ τ then β τ ← 0. else ∀j ∈ [p] β τ j ←        ∥ β∥ ∞ -τ if ∥ β∥ ∞ -2τ ≤ β j and β j ≥ 0, -∥ β∥ ∞ + τ if β j ≤ -∥ β∥ ∞ + 2τ and β j < 0, β j otherwise.
end if return β τ

Full characterization of the uniform uniqueness

For the pattern recovery by the τ -thresholded penalized estimator, in Theorem 5.5.1 we assume the uniform uniqueness. For that reason, below we provide a a necessary and sufficient condition for uniform uniqueness of the penalized optimization problem (5.1.1) in Theorem 5.6.1. This theorem extends the Theorem 1 from [START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF] to all polyhedral gauges. Then the solution to the above problem is unique, i.e, S X,λpen (y) is a singleton for all y ∈ R n if and only if row(X) does not intersect a face of the polytope B * = conv{0, u 1 , . . . , u l } whose dimension is smaller than dim(ker(X)).

Note that every face F of B * satisfies dim(F ) < dim(ker(X)) ⇐⇒ codim(F ) > rk(X),

where codim(F ) = p -dim(F ). For a better explanation of the non-uniqueness of the estimator, below we give an example for the generalized LASSO with pen(b) = ∥Db∥ 1 . Note that if ker(X) ∩ ker(D) = {0}, then for every β ∈ S X,λ∥D.∥ 1 (y) and g ∈ ker(X) ∩ ker(D) we have β + g ∈ S X,λ∥D.∥ 1 (y), too. Thus for every y ∈ R n the minimizer is not unique. Consequently, ker(X) ∩ ker(D) = {0} is a necessary condition for uniform uniqueness, yet, it is not sufficient, as illustrated in the example given in [START_REF] Barbara | Maximal solutions of sparse analysis regularization[END_REF], which we revisit below.

∈ R p 1 2 ∥Y -Xb∥ 2 2 + 1 2 ∥Db∥ 1 ,
where

X =    1 1 1 3 1 1 √ 2 0 0    , D =    1 1 0 1 0 1 2 1 1    and Y =    1 1 0    .
We have S X, 1 2 ∥D.∥ 1 (Y ) = conv{(0, 1/2, 0) ′ , (0, 0, 1/2) ′ } [9, p. 19]. Since ∥Db∥ 1 = max{±(4b 1 + 2b 2 + 2b 3 ), ±(2b 1 + 2b 2 ), ±(2b 1 + 2b 3 )} then B * = conv{±(4, 2, 2) ′ , ±(2, 2, 0) ′ , ±(2, 0, 2) ′ }. Because the vertex F = (4, 2, 2) ′ is a face of B * which lies in row(X) and satisfies dim(F ) = 0 < 1 = dim(ker(X)) then, according to Theorem 5.6.1, the uniform uniqueness cannot hold. This complies with the fact that S X, 1 2 ∥D.∥ 1 (Y ) is not a singleton.

When ker(X) ∩ ker(D) = {0}, in broad generality, the set of generalized LASSO solutions is a polytope (a bounded polyhedron) [9, Proposition 4.3.] and its extremal points can be explicitly computed [START_REF] Dupuis | The geometry of sparse analysis regularization[END_REF]. This description is relevant when this set is not a singleton. Lemma 5.6.1. Consider the function

f (b) = 1 2 ∥Y -Xb∥ 2 2 + pen(b),
where pen(b) = max{0, u ′ 1 b, . . . , u ′ l b} is a polyhedral gauge. Then the condition ker(X) ∩ ker(D) = {0} is necessary for the uniform uniqueness of the minimizer of f . Proof. Let 0 ̸ = h ∈ ker(X) ∩ ker(pen) and let β ∈ S X,λpen (Y ). Then

f ( β) ≤ f ( β + h) = 1 2 ∥Y -X( β + h)∥ 2 2 + pen( β + h) = 1 2 ∥Y -X β∥ 2 2 + max{0, u ′ 1 ( β + h), . . . , u ′ l ( β + h)}.
Since pen(h) = 0, we have u ′ i h ≤ 0 for every i ∈ [l], hence

f ( β + h) ≤ 1 2 ∥Y -X β∥ 2 2 + max{0, u ′ 1 β, . . . , u ′ l β} = f ( β).
Therefore β is not a unique minimizer of f .

Numerical experiments

Below, in our simulations, we consider the linear regression model Y = Xβ + ε where:

• The matrix X = (X 1 | . . . |X 150 ) ∈ R 100×150 has i.i.d. N (0, 1/100) entries.

• The random noise ε ∈ R n has i.i.d. N (0, 1) entries.

Numerical experiments for LASSO

For LASSO, the noiseless recovery condition and the accessibility condition depend on β through sign(β) ∈ {-1, 0, 1} p . Moreover, since the distribution of X is invariant by permutations and sign changes of the columns of X, then the probability that a k-sparse vector satisfies the noiseless recovery condition is equivalent to the irrepresentability condition: • LASSO with a large tuning parameter λ = 2 2 log(150) (as suggested by Candès and Plan [START_REF] Candès | Near-ideal model selection by ℓ 1 minimization[END_REF]).

P X ∥X ′ (X ′ I ) + 1 k ∥ ∞ ≤
• LASSO with a small tuning parameter; the one provided by SURE formula, which for a given X and Y minimizes the function 0

< λ → 1 2 ∥Y -X β(λ)∥ 2 2 + |{i ∈ [p] : β i (λ) ̸ = 0}|
where β(λ) is the LASSO estimator (see e.g. [START_REF] Tibshirani | Degrees of freedom in lasso problems[END_REF] or [START_REF] Vaiter | The degrees of freedom of partly smooth regularizers[END_REF]).

Numerical experiments when the penalty term is the supremum norm

For the ℓ ∞ regularization, the noiseless recovery condition and the accessibility condition depend on β through sign ∞ (β) ∈ {-1, * , 1} p . Same as for LASSO, since the distribution of X is invariant by permutations and sign changes of the columns of X, then the probability that a non-zero vector having k non-maximal components in absolute value satisfies the noiseless recovery condition is given by

P X ( X ′ ( X ′ ) + e 1 = e 1 ) where X = ( X 1 |X I ) with X 1 = p-k i=1 X i and I = {p -k + 1, . . . , p}.
An explicit formula for checking the noiseless recovery condition is given in the Appendix. Moreover, the accessibility condition is satisfied with probability P X (min{∥γ∥ ∞ : Xγ = X 1 } = 1). 150) the sign of β is recovered both by LASSO and thresholded LASSO (top left). When the tuning parameter is small (computed by SURE), some null components of β are not correctly estimated at 0 (black points outside the x-axis), but there exists a threshold, for which the thresholded LASSO recovers the sign of β (top right). On the bottom, when k = 30, the accessibility condition holds but the noiseless recovery condition does not hold, thus thresholded LASSO can recover the sign of β but LASSO cannot. When the tuning parameter is large: λ = 2 2 log(150), both LASSO and thresholded LASSO fail to recover the sign of β (bottom left). When the tuning parameter is small, some null components of β are not correctly estimated at 0, but there exists a threshold, for which the thresholded LASSO recovers the sign of β (bottom right).

< λ → 1 2 ∥Y -X β(λ)∥ 2 2 + |{i ∈ [p] : | β i | < ∥ β(λ)∥ ∞ }|
, where β(λ) is the unique element of S X,λpen (Y ) (see e.g. [START_REF] Minami | Degrees of freedom in submodular regularization: A computational perspective of Stein's unbiased risk estimate[END_REF] or [START_REF] Vaiter | The degrees of freedom of partly smooth regularizers[END_REF]). Then the subdifferential of ϕ at x is a face of P and is given by

∂ϕ(x) = conv{v l : l ∈ I ϕ (x)} = {s ∈ P : s ′ x = ϕ(x)}, where I ϕ (x) = {l ∈ [k] : v ′ l x = ϕ(x)}.

Proofs

Proof of Theorem 5.1.1

For γ ≥ 0, as P γ we denote the polyhedron: Let us revisit the polyhedron P γ . As every polyhedron, P γ can be decomposed as the sum of its recession cone and a bounded polyhedron [START_REF] Ziegler | Lectures on Polytopes[END_REF]Theorem 1.2. and Proposition 1.12.]. Therefore, for γ = 1 we have:

P γ = {b ∈ R p : pen(b) ≤ γ} = {b ∈ R p : u ′ 1 b ≤ γ, . . . , u ′ l b ≤ γ}.
P 1 = {b ∈ R p : u ′ 1 b ≤ 0, . . . , u ′ l b ≤ 0} + E
, where E is a polytope. Note that, for an arbitrary γ ≥ 0, we have P γ = P 0 + γE.

(β k ) k≥1 that f (β k ) ≤ m + δ k ≤ f (0) for every k ≥ 1.
Step 1: Convergence of (Xβ k ) k≥1 and (pen(β k )) k≥1 : At first we show that (Xβ k ) k≥1 is bounded. Indeed, we have

0 ≤ 1 2 ∥Y -Xβ k ∥ 2 2 ≤ f (β k ) ≤ 1 2 ∥Y ∥ 2 2 .
Thus

∥Xβ k ∥ 2 ≤ ∥Y ∥ 2 +∥Xβ k -Y ∥ 2 ≤ 2∥Y ∥ 2 and the sequence (Xβ k ) k≥1 is bounded. Thus, if
it is not convergent, then there exist two subsequences (Xβ r k ) k≥1 k→∞ -→ r and (Xβ s k ) k≥1 k→∞ -→ s converging to two different limits. By the strict convexity of φ(t) := ∥Y -t∥ 2 2 , one may deduce the following inequality:

lim k→∞ ∥Y -X β r k + β s k 2 ∥ 2 2 = ∥Y - r + s 2 ∥ 2 2 < 1 2 ∥Y -r∥ 2 2 + 1 2 ∥Y -s∥ 2 2 .
Moreover, the convexity of pen yields

pen β r k + β s k 2 ≤ 1 2 pen(β r k ) + pen(β s k ) . Consequently, lim sup k→∞ f β r k + β s k 2 < 1 2 f (β r k ) + f (β s k ) = m,
= inf b ∈ R p f (b) = 1 2 ∥Y -g∥ 2 2 + λγ.
Let β be an arbitrary solution of (5.8.1) with R = γ. We are going to prove that f

( β) = 1 2 ∥Y -g∥ 2 2 + λγ = m. Case 1: If γ > 0, then for k large enough such that pen(β k ) > 0 one may set v k := γ pen(β k ) β k . We have pen(v k ) = γ and thus v k ∈ P γ . Consequently, by definition of β, we have ∥Y -Xv k ∥ 2 2 ≥ ∥Y -Xβ k ∥ 2 2 . Therefore f ( β) = 1 2 ∥Y -X β∥ 2 2 + λpen( β) ≤ 1 2 ∥Y -Xv k ∥ 2 2 + λγ k→∞ -→ ∥Y -g∥ 2 2 + λγ = m.
Case 2: Let γ = 0. Because P 1 = P 0 + E, where E is a bounded polyhedron, one may write β k = v k + pen(β k )w k where v k ∈ P 0 and w k ∈ E. Because Xβ k → g and pen(β k )w k → 0, one may deduce that Xv k → g. Moreover, since pen(β k )v k ∈ P 0 , we have the following inequality:

f ( β) = 1 2 Y -X β 2 2 ≤ 1 2 ∥Y -Xv k ∥ 2 2 k→+∞ -→ 1 2 ∥Y -g∥ 2 2 = m,
which achieves the proof.

Proof of Theorem 5.6.1

Proof of Theorem 5.6.1. ( =⇒ ) Assume that there exists a face F of B * = conv{0, u 1 , . . . , u l } that intersects row(X) and satisfies dim(F ) < dim(ker(X)). By Lemma 5.8.1, F = ∂pen( β) for some β ∈ R p . Let z ∈ R n with X ′ z ∈ F , which exists by assumption. Now let y = X β + λz.

Note that β ∈ S X,λpen (y) since l β < pen( β) whenever l / ∈ I. Now we show that it is possible to pick h ∈ ker(X) with h ̸ = 0, but u ′ l h = 0 for all l ∈ I. Then we can make h small enough such that u ′ l ( β + h) ≤ pen( β) still holds for all l / ∈ I, which in turn implies that pen(

0 ∈ ∂ 1 2 ∥y -X β∥ 2 2 + λpen( β) = X ′ X β -X ′ y + λ∂pen( β) (5.8.2) ⇐⇒ 1 λ X ′ (y -X β) = X ′ z ∈ ∂pen( β). ( 5
β + h) = max{u ′ l β : l ∈ I} = pen( β). This, together with X β = X( β + h), yields β ̸ = β = β + h ∈ S X,λpen (y). The existence of 0 ̸ = h ∈ ker(X) such that u ′ l h = 0 for every l ∈ I is equivalent to ker(X) ∩ (col(U )) ⊥ ̸ = {0},
where U = (u l ) l∈I ∈ R p×|I| . To prove it, we distinguish two cases:

1) Assume that 0 ∈ aff{u l : l ∈ I}. Then aff{u l : l ∈ I} = col(U ) and rk(U ) = dim(F ) < dim(ker(X)). This implies that dim(ker(X)) + dim((col(U )) ⊥ ) > p, Lemma 5.8.8. Let X ∈ R n×p , λ > 0 and pen be a polyhedral gauge on R p . Assume that the uniform uniqueness holds for (5.1.1). Let β ∈ R p , ε ∈ R n and set y (r) = X(rβ) + ε.

If β is accessible with respect to X and pen, then lim r→∞ β(y (r) )/r = β.

Proof. Since β(y (r) ) ∈ S X,λpen (y (r) ), the following inequality holds

1 2 ∥y (r) -X β(y (r) )∥ 2 2 + λpen( β(y (r) )) ≤ 1 2 ∥y (r) -X(rβ)∥ 2 2 + λpen(rβ). y (r) -X(rβ) = ε, therefore one may deduce that 1 2 ∥ε + X(rβ) -X β(y (r) )∥ 2 2 + λpen( β(y (r) )) ≤ 1 2 ∥ε∥ 2 2 + λpen(rβ) (5.8.5)
and in particular

λpen( β(y (r) )) ≤ 1 2 ∥ε∥ 2 2 + λpen(rβ) =⇒ pen( β(y (r) )/r) ≤ ∥ε∥ 2 2 2λr + pen(β) =⇒ lim sup r→∞ pen( β(y (r) )/r) ≤ pen(β). (5.8.6)
Consequently, the sequence pen( β(y (r) )/r) r∈N + is bounded. By the Cauchy-Schwarz inequality, the inequality (5.8.5) implies that

-∥ε∥ 2 ∥X(rβ) -X β(y (r) )∥ 2 + 1 2 ∥X(rβ) -X β(y (r) )∥ 2 2 ≤ λpen(rβ) -λpen( β(y (r) )) (5.8.7) =⇒ - ∥ε∥ 2 r X β(y (r) ) r -β 2 + 1 2 X β(y (r) ) r -β 2 2 ≤ λpen(β) r - λ r pen β(y (r) ) r .
(

(r) )/r) r ≥ α 2 /2 if α < ∞ ∞ if α = ∞.
Moreover, since the sequence pen( β(y (r) )/r) We conclude that α = 0 and the sequence (5.8.9) converges to 0.

Noiseless recovery condition for the supremum norm

Recall the noiseless pattern recovery for the supremum norm:

∃λ > 0 ∃ β ∈ S X,λ∥.∥∞ (Xβ) such that β ∥.∥∞ ∼ β.
Remark 5.8.1. Let pen ≥ 0 be minimized at 0. Then β = 0 minimizes the function

f (b) = 1 2 ∥Xb∥ 2 2 + λpen(b).
Proof. The proof is a direct consequence of 0 minimizing both ∥Xb∥ 2 and pen(b).

|β i | < ∥β∥ ∞ }. Let X = ( X1 |X I ), where X1 = i / ∈I sign(β i )X i .
Then the noiseless pattern recovery occurs if and only if

X′ ( X′ ) + e 1 = e 1 ,
where e 1 = (1, 0, . . . , 0) ′ .

Before giving the proof, we recall that the subdifferential of the ℓ ∞ norm at 0 is the unit ball of the ℓ 1 norm and for x ̸ = 0 this subdifferential is equal to

∂∥.∥ ∞ (x) = s ∈ R p : ∥s∥ 1 ≤ 1 and s ′ x = ∥x∥ ∞ = s ∈ R p : ∥s∥ 1 = 1 and s i x i ≥ 0 if |x i | = ∥x∥ ∞ s i = 0 otherwise . ( 5
λ X ′ X(β -β) ∈ ∂∥.∥ ∞ (β) = ∂∥.∥ ∞ ( β).
(5.8.11)

Let us set c = (∥β∥ ∞ , β ′ I ) ′ and ĉ = (∥ β∥ ∞ , β′ I ) ′ . By construction, Xβ = Xc and X β = Xĉ. Consequently, according to (5.8.11), we have

1 λ X ′ X(c -ĉ) ∈ ∂∥.∥ ∞ (β).
(5.8.12)

By the representation (5.8.10) of ∂∥ • ∥ ∞ (β), the relation above implies that

   ∀i ∈ I, X ′ i X(c -ĉ) = 0, 1 λ i / ∈I sign(β i )X ′ i X(c -ĉ) = 1, therefore 1 λ X′ X(c -ĉ) = e 1 ⇒ X(c -ĉ) = λ( X′ ) + e 1 .
Using the last implication and (5.8.12), one may deduce that

X ′ ( X′ ) + e 1 = 1 λ X ′ X(c -ĉ) ∈ ∂∥ • ∥ ∞ (β).
Finally, we prove that X′ ( X′ ) + e 1 = e 1 :

X ′ ( X′ ) + e 1 ∈ ∂∥.∥ ∞ (β) ⇒    ∀i ∈ I X ′ i ( X′ ) + e 1 = 0 i / ∈I sign(β i )X ′ i ( X′ ) + e 1 = X ′ 1 ( X′ ) + e 1 = 1 ⇒ X′ ( X′ ) + e 1 = e 1 .
(⇐=) Again, let c = (∥β∥ ∞ , β ′ I ) ′ . Denote ĉ = c -λ X+ ( X′ ) + e 1 and set β as follows:

∀i ∈ [p], βi := ĉi when i ∈ I, sign(β i )ĉ 1 when i / ∈ I.

Note that ∥ β∥ ∞ = c 1 = ∥β∥ ∞ . Moreover, by definition of I, c 1 is the unique coordinate of c having the maximal absolute value. Therefore, for λ small enough, we have ĉ1 = ∥ĉ∥ ∞ . It implies that ∥ β∥ ∞ = ĉ1 and this value is attained exactly at coordinates βi : i /

∈ I. Therefore {i ∈ [p] : | βi | < ∥ β∥ ∞ } = {i ∈ [p] : |β i | < ∥β∥ ∞ } = I. Moreover, for i / ∈ I we have β i βi = β i sign(β i )ĉ 1 = ∥β∥ ∞ ĉ1 > 0. Consequently, for λ small enough we have sign ∞ ( β) = sign ∞ (β).
To conclude the proof it is enough to show that β ∈ S X,λ∥.∥∞ (Xβ), i.e. 1 λ X ′ (Xβ -X β) ∈ ∂∥.∥ ∞ ( β). By Remark 2.3.9 X X+ is the projection onto col( X) and col(( X′ ) + ) = col( X). Therefore

1 λ X ′ X(β -β) = 1 λ X ′ ( Xc -Xĉ) = X ′ X X+ ( X′ ) + e 1 = X ′ ( X′ ) + e 1 .
Then it suffices to prove that the latter term belongs to ∂∥.∥ ∞ (β). As we assumed that X ′ ( X′ ) + e 1 = e 1 , we have

X′ ( X′ ) + e 1 = e 1 ⇒ X ′ 1 ( X′ ) + e 1 = 1 ∀i ∈ I X ′ i ( X′ ) + e 1 = 0 ⇒    i / ∈I sign(β i )X ′ i ( X′ ) + e 1 = 1 ∀i ∈ I X ′ i ( X′ ) + e 1 = 0 ⇒ X ′ ( X′ ) + e 1 ∈ ∂∥.∥ ∞ (β) .
Consequently, for λ > 0 small enough, β ∈ S X,λ∥.∥∞ (Xβ) and sign ∞ ( β) = sign ∞ (β).

Introduction and preliminaries

The content of this chapter comes from the published paper of Bogdan, Bosy and the author of the dissertation [START_REF] Bogdan | Maximum likelihood estimation for discrete exponential families and random graphs[END_REF]. Exponential families are of paramount importance in probability and statistics. They were introduced by Fisher, Pitman, Darmois and Koopman in 1934-36 and have many properties that make them indispensable in theory and applications, see [START_REF] Lehmann | Theory of point estimation[END_REF] (Section 2.7), [START_REF] Barndorff-Nielsen | Information and exponential families in statistical theory[END_REF] (Chapter 9), [START_REF] Andersen | Sufficiency and exponential families for discrete sample spaces[END_REF], [START_REF] Diaconis | Group representations in probability and statistics[END_REF] (Chapter 9.E), [START_REF] Diaconis | Partial exchangeability and sufficiency[END_REF], and [START_REF] Lauritzen | Extreme point models in statistics[END_REF]. In this paper we study discrete exponential families, more specifically, exponential families on finite sets, and give a new characterization of the existence of the maximum likelihood estimator (MLE) for exponential family and the data at hand. We also present applications, in particular for specific exponential families we give a threshold of the sample size sufficient for the existence of MLE with high probability for i.i.d. samples.

The computation of MLE is in general difficult with the number of variables increasing. On the other hand, for given data and an exponential family, MLE may fail to exist. In particular, [START_REF] Crain | Estimation of distributions using orthogonal expansions[END_REF][START_REF] Crain | Exponential models, maximum likelihood estimation, and the Haar condition[END_REF] pointed out to problems with the maximum likelihood estimation when the number of parameters is too large for the sample size. He also gave a sufficient condition for MLE to exist almost surely -the Haar condition.

A complete characterization of the existence of MLE for rather general exponential families was given by Barndorff-Nielsen. Namely, by [START_REF] Barndorff-Nielsen | Information and exponential families in statistical theory[END_REF] (Theorem 9.13), MLE for a sample and an exponential family exists if and only if the vector of the sample means calculated for a basis of the linear space of exponents belongs to the interior of the convex hull of the pointwise range of the basis. This beautiful criterion is alas cumbersome to apply. Therefore, [START_REF] Jacobsen | Existence and unicity of MLEs in discrete exponential family distributions[END_REF] gives an alternative condition for discrete exponential families, together with applications to Cox regression, logistic regression and multiplicative Poisson models. Similar condition is presented by [START_REF] Albert | On the existence of maximum likelihood estimates in logistic regression models[END_REF] for loglinear model. [START_REF] Haberman | The analysis of frequency data[END_REF] gives a characterization of the existence of MLE for hierarchical log-linear models. His conditions can be interpreted in terms of polytope geometry, see also [START_REF] Eriksson | Polyhedral conditions for the nonexistence of the MLE for hierarchical log-linear models[END_REF], and [START_REF] Fienberg | Maximum likelihood estimation in log-linear models[END_REF]. [START_REF] Brown | Fundamentals of statistical exponential families with applications in statistical decision theory[END_REF] characterizes the existence of MLE when the log-partition function is steep and regularly convex, and interprets the problem of finding MLE as the optimization of the Kullback-Leibler 105 divergence. [START_REF] Darroch | Markov fields and log-linear interaction models for contingency tables[END_REF] connect the properties of MLE in decomposable models with graph-theoretical notions, thus starting the theory of graphical models in statistics. Sufficient conditions for the existence of MLE in specific exponential families are also given by [START_REF] Stone | Large-sample inference for log-spline models[END_REF] and [START_REF] Bogdan | Testing uniformity via log-spline modeling[END_REF]. [START_REF] Geyer | Likelihood and exponential families[END_REF] looks for MLE in the closure of convex exponential families, relates the existence of MLE with the linear programming feasibility problem, and in the case of nonexistent MLE, reduces the considered exponential family until MLE exists for the family. He also applies MCMC algorithms to calculate MLE. A comparison between the conditions of Barndorff-Nielsen and Jacobsen is discussed in [START_REF] Konis | Linear programming algorithms for detecting separated data in binary logistic regression models[END_REF]. In addition, Konis presents an implementation of Jacobsen's test using linear programming. A broad survey of the history of log-linear models and further motivation for the study of the existence of MLE can be found in [START_REF] Fienberg | Three centuries of categorical data analysis: log-linear models and maximum likelihood estimation[END_REF][START_REF] Fienberg | Maximum likelihood estimation in log-linear models[END_REF].

The main inspiration for our work is [START_REF] Bogdan | On existence of maximum likelihood estimators in exponential families[END_REF] (Theorem 2.3) on the existence of MLE for exponential families of continuous functions on finite interval. In Theorem 6.2.2 below we propose a similar characterization, which is new in the setting of discrete exponential families. We obtain the result by a straightforward, self-contained approach, which does not depend on the delicate convex analysis of [START_REF] Barndorff-Nielsen | Information and exponential families in statistical theory[END_REF].

The paper is composed as follows. In Section 6.2 we state and prove our criterion, using the notion of set of uniqueness. The criterion is restated in Section 6.2.2 as a linear programming problem. In Section 6.3 we give applications to exponential families spanned by Rademacher and Walsh functions, and to exponential families of random graphs. In particular we give sharp or plain thresholds for the sample size to secure the existence of MLE with high probability. In Appendix 6.6 we give auxiliary results and reformulations of our criterion and pin down its connections with the criterion of Barndorff-Nielsen.

Discrete exponential family

Consider a finite set X ̸ = ∅ and weight function µ : X → (0, ∞). As usual, R X is the family of all the real-valued functions on X . For ϕ ∈ R X we define the partition and the log-partition Moreover, for ϕ 1 , ϕ 2 ∈ R X we have e(ϕ 1 ) = e(ϕ 2 ) if and only if ϕ 1 -ϕ 2 is constant. Consider x 1 , . . . , x n ∈ X , a sample. For ϕ ∈ R X we denote, as usual,

functions, Z(ϕ) = x∈X e ϕ(x) µ(x), ψ(ϕ) = log Z(ϕ), ( 6
φ = 1 n n i=1 ϕ (x i ).
The likelihood function of p = e(ϕ) is defined as We note that the likelihood functions are uniformly bounded. Indeed, for every ϕ ∈ R X , Since X is a finite set, e(B) will be called discrete exponential family (we do not consider infinite countable sets, for which see [START_REF] Jacobsen | Existence and unicity of MLEs in discrete exponential family distributions[END_REF]). We call p ∈ e(B) an MLE for x 1 , . . . , x n and e(B) if

L e(ϕ) (x 1 , . . . , x n ) = L p (x 1 , . . . , x n ) = n i=1 p(x i ), 6
ψ(ϕ) = log x∈X e ϕ(x) µ(x) ≥ max X ϕ + min X log µ, ( 6
L p (x 1 , . . . , x n ) = sup p∈e(B) L p (x 1 , . . . , x n ) ,
or, equivalently,

l p (x 1 , . . . , x n ) = sup p∈e(B) l p (x 1 , . . . , x n ) .
The following result is well known (see, e.g., [START_REF] Johansen | Introduction to the theory of regular exponential families[END_REF] (Theorem 2.1) or [START_REF] Diaconis | Group representations in probability and statistics[END_REF] (p. 177)), but for the reader's convenience we give a proof in Appendix 6.6.1. We note that the first supremum in Example 6.1.1 is approached when b → ∞, or for the density p = 1 {1} , which, however, is not in e(R X ) but rather in e(R {1} ). Below in Theorem 6.2.2 we characterize the situation when the genuine MLE exists, and in Theorem 6.2.5 we treat, by a suitable reduction of X , the case when the supremum of the likelihood function is ''attained at infinity". Before we proceed, we owe the reader some comments on the notation used in this paper and in the literature.

Alternative setting

Let d be a natural number. Consider a nonempty finite set S ⊂ R d , weight m on S and the linear space spanned by the coordinate functions on R d . The corresponding exponential densities have the form π θ (y) = e θ•y /ζ(θ), y ∈ S, (6.1.9)

where θ ∈ R d , • is the scalar product in R d and ζ(θ) = y∈S e θ•y m(y). Thus, (6.1.9) is a natural, or standard, exponential family, see [START_REF] Letac | Lectures on natural exponential families and their variance functions, volume 50 of Monografías de Matemática[END_REF] or [START_REF] Brown | Fundamentals of statistical exponential families with applications in statistical decision theory[END_REF]. Since the range of the vector of parameters θ is the whole of R d , which is open, the exponential family (6.1.9) is regular, see [START_REF] Lauritzen | Graphical models, volume 17 of Oxford Statistical Science Series[END_REF] (Appendix D.1).

The setting is actually generic, as we explain momentarily. If functions ϕ 1 , . . . , ϕ d span the linear space B in the general discussion above and we let T (x) = (ϕ 1 (x), . . . , ϕ d (x)) for x ∈ X , then for every ϕ ∈ B there is θ ∈ R d such that ϕ(x) = θ • T (x) for x ∈ X , and

e(ϕ) = e θ•T /Z(θ • T ). (6.1.10)
This is the form used by most authors, see [START_REF] Lauritzen | Graphical models, volume 17 of Oxford Statistical Science Series[END_REF] or [START_REF] Johansen | Introduction to the theory of regular exponential families[END_REF], and T is called the canonical statistics.

Furthermore, we let S = T (X ) ⊂ R d and m(y) = x:T (x)=y µ(x) for y ∈ S. With the notation of (6.1.9) and (6.1.10) we have and {e(ϕ) : ϕ ∈ B}. This makes a complete connection between our setting and the setting of natural exponential families with finite support S. The same setting of discrete exponential families on finite set is described, using slightly different language, in [START_REF] Sullivant | Algebraic statistics[END_REF] ( §6.2). We also recall that if ϕ 1 , . . . , ϕ d are affinely independent, then the representation (6.1.10) is minimal, see [START_REF] Johansen | Introduction to the theory of regular exponential families[END_REF] (Chapter 1) or [START_REF] Lauritzen | Graphical models, volume 17 of Oxford Statistical Science Series[END_REF], where the affine independence means that θ • T = const implies θ = 0. In general, one allows the representation to be nonminimal because over-parametrization is often natural in applications. We shall return to this discussion again in Section 6.6.6, but for now we get back to the setting of B and (6.1.8). The latter allows to work without coordinates and benefit from properties of specific linear spaces B, which could otherwise be obscured by an arbitrary choice of T and S.

π θ (y) = e(ϕ)(x) if T (x) = y. ( 6

Main results

Let 1 denote the function on X identically equal to 1. Assume that 1 ∈ B. This entails no restriction on the considered exponential families e(B), but allows an elegant formulation of the criterion of existence of MLE in terms of B, in fact in terms of the cone of nonnegative functions in B:

B + := {ϕ ∈ B : ϕ ≥ 0}.
We note in passing that Appendix 6.6.6 gives a reformulation of our criterion for the existence of MLE without requiring that 1 ∈ B.

Let U ⊂ X . We say that U is a set of uniqueness for B if ϕ = 0 is the only function in B such that ϕ = 0 on U . Similarly, we say that U is a set of uniqueness for B + if ϕ = 0 is the only function in B + such that ϕ = 0 on U . Put differently, U is of uniqueness for B + if the conditions ϕ ∈ B + and ϕ = 0 on U imply that ϕ = 0 on X . Of course, if U is a set of uniqueness for B, then U is a set of uniqueness for B + .

Nonexistence of MLE

In this section we elaborate on the case of nonexistence of MLE in the spirit of [START_REF] Geyer | Likelihood and exponential families[END_REF]. To this end we fix x 1 , . . . , x n ∈ X and assume that there is a nontrivial δ ∈ B + such that δ(x 1 ) = . . . = δ(x n ) = 0. By Theorem 6.2.2, sup p∈e(B) l p (x 1 , . . . , x n ) is not attained at any p ∈ e(B). However, the supremum is "attained at infinity", in fact for an exponential density on a proper subset of the state space X . Indeed, fix δ as above. If φ ∈ B and k ∈ (0, ∞), then

l e(φ) (x 1 , . . . , x n ) ≤ l e(φ-kδ) (x 1 , . . . , x n ),
see the first part of the proof of Theorem 6.2.2. Furthermore,

ψ (φ -kδ) → log x∈X :δ(x)=0 e φ(x) µ(x), as k → ∞. (6.2.1)
We let X = {x ∈ X : δ(x) = 0} and carrying on with the notation for X we obtain measure µ, linear space B with cone B + , log-partition function ψ, likelihood function L, log-likelihood function l and exponential family e( B). Put simpler, we discard {x ∈ X : δ(x) > 0} and achieve the following reduction.

Lemma 6.2.1. sup p∈e( B) l p (x 1 , . . . , x n ) = sup p∈e(B) l p (x 1 , . . . , x n ). Proof. For ϕ ∈ B we let ϕ = ϕ| X . Since {x 1 , . . . , x n } ⊂ X , ϕ = 1 n n i=1 ϕ(x i ) = 1 n n i=1 ϕ(x i ) = ϕ. ( 6.2.2)
Furthermore,

ψ(ϕ) = log x∈X e ϕ(x) µ(x) ≥ log   x∈ X e ϕ(x) µ(x)   = ψ( ϕ). Thus ϕ -ψ(ϕ) ≤ ϕ -ψ( ϕ),
-2} B + = {-2} and {-1} B + = {-2, -1, 0}.
We note that if x ̸ ∈ {x 1 , . . . , x n } B + , then there is ϕ ∈ B + such that ϕ = 0 on {x 1 , . . . , x n } but ϕ(x) > 0. Since X is finite, by adding such functions we can construct δ ∈ B + that vanishes precisely on {x 1 , . . . , x n } B + , i.e., δ -1 ({0}) = {x 1 , . . . , x n } B + . We adopt the setting of Lemma 6.2.1 with this δ, in particular with X = {x 1 , . . . , x n } B + , and we get the following result. Theorem 6.2.5. There is a unique p ∈ e( B) such that l p (x 1 , . . . , x n ) = sup p∈e(B) l p (x 1 , . . . , x n ).

Proof. By the definition of {x 1 , . . . , x n } B + and by Theorem 6.2.2, Lemma 6.1.1 and 6.2.1, there is a unique p ∈ e( B) such that

l p (x 1 , . . . , x n ) = sup p∈e( B) l p(x 1 , . . . , x n ) = sup p∈e(B) l p (x 1 , . . . , x n ) .
Example 6.2.6. For the first sample in Example 6.1.1 we get X = {x 1 } B + = {1}, and p = 1 on X .

For more substantial applications of Theorem 6.2.5 we refer to Example 6.3.1 and Example 6.3.5.

Linear programming

Before we address special spaces B, we offer the reader a down-to-earth perspective. To start with, by a comment at the beginning of Section 6.2, we get the following simple result. Notably, the condition in Corollary 6.2.1 may be verified by solving the following linear problem:

ϕ ∈ B, ϕ(x 1 ) = ... = ϕ(x n ) = 0.
Indeed, {x 1 , . . . , x n } is of uniqueness for B if and only if the homogeneous linear system has only the trivial solution. In contrast, Theorem 6.2.2 is a linear programming problem. Indeed, {x 1 , . . . , x n } is of uniqueness for B + if and only if the supremum of the (objective) function x∈X ϕ(x) is zero for the class of functions satisfying

ϕ ∈ B, ϕ(x 1 ) = ... = ϕ(x n ) = 0, ϕ ≥ 0.
In this vein [START_REF] Rinaldo | On the geometry of discrete exponential families with application to exponential random graph models[END_REF] (Appendix C) observe that the condition of Barndorff-Nielsen is actually a linear programming problem and make connections to the geometry (of the convex hull of the set S in Section 6.1.2). The linear programming also occurs in the study of the closures of convex exponential families [START_REF] Geyer | Likelihood and exponential families[END_REF] or binary logistic regression models [START_REF] Konis | Linear programming algorithms for detecting separated data in binary logistic regression models[END_REF]. Furthermore, [START_REF] Wang | Approximating faces of marginal polytopes in discrete hierarchical models[END_REF] consider the linear programming in the case when MLE fails to exist. See also [START_REF] Sullivant | Algebraic statistics[END_REF] for further information on linear programming and cases of nonexistence of MLE for discrete exponential families. Since the linear programming in general runs in polynomial time, see [START_REF] Schrijver | Theory of linear and integer programming[END_REF], it should be the method of choice when verifying the existence of MLE for discrete exponential families Proof. By Theorem 6.2.2 we only need to prove that the above condition characterizes the sets of uniqueness for B k + . If j ∈ {1, . . . , k} is such that r j (x 1 ) = . . . = r j (x n ) = 1, then we let r = r 0 -r j . Clearly, r ∈ B k + and r is not identically zero, but r(x i ) = 0 for all i = 1, . . . , n. Thus, {x 1 , . . . , x n } is not a set of uniqueness for B k + . Similarly, if r j (x 1 ) = . . . = r j (x n ) = -1, then we consider the function r = r 0 + r j ∈ B k + . For the converse implication we consider arbitrary Assume that r = 0 on {x 1 , . . . , x n }. Let j ∈ {1, . . . , k}. There are x, x ′ ∈ {x 1 , . . . , x n } such that r j (x) = 1 and r j (x ′ ) = -1. We have

0 = r(x) + r(x ′ ) = 2a 0 + i̸ =j a i [r i (x) + r i (x ′ )].
It follows that a 0 ≤ i̸ =j

|a i |.
By (6.3.2), a j = 0, for every j ≥ 1. Thereby a 0 = 0 and r ≡ 0. We see that {x We define the positive and negative half-cubes, respectively:

H + j = {χ ∈ Q k : r j (χ) = 1}
, H - j = {χ ∈ Q k : r j (χ) = -1}, j = 1, . . . , k. (6.3.3) We note that B k is also spanned by the indicator functions of half-cubes, namely 1 + j = (r 0 +r j )/2 and 1 - j = (r 0 -r j )/2, j = 1, . . . , k. Since we consider the case when MLE does not exist, by Theorem 6.3.3, J ′ ̸ = ∅. For j ∈ J ′ we let H j = {χ ∈ Q k : r j (χ) = r j (x 1 ) = . . . = r j (x n )}.

Proof.

Observe that ν uniq = max {τ 1 , . . . , τ k }, where τ j = min {n ≥ 1 : {r j (X 1 ), . . . , r j (X n )} = {-1, 1}} , j = 1, . . . , k.

From the fact that X 1 , X 2 , . . . are independent and uniformly distributed, we deduce that 1 r j (X i )̸ =r j (X 1 ) , i = 2, 3, . . . , j = 1, 2 . . . , are independent with symmetric Bernoulli distribution. Then τ 1 , . . . , τ k are independent, and

τ j + 1 ∼ Geom (1/2)
for j = 1, . . . , k. The result follows from [START_REF] Eisenberg | On the expectation of the maximum of IID geometric random variables[END_REF].

In Section 6.5 we return to Rademacher functions, but for now we turn to exponential families of random graphs, a major motivation for this work.

are the Walsh functions, see, e.g., [START_REF] Jendrej | On some extensions of the FKN theorem[END_REF].

The case B k 1 = B k was discussed in Section 6.3.1 and the case q = 2 is related to the Ising model of ferromagnetism in statistical mechanics, see [START_REF] Wainwright | Graphical models, exponential families, and variational inference[END_REF] (Example 3.1). Lemma 6.5.1. The dimension of the linear space B k q is q j=0 k j . The proof of Lemma 6.5.1 is given in Appendix 6.6.5. Corollary 6.5.1. For q ≤ k 2 we have

dim B k q ≤ 2 kH 2 ( q k ) ≤ ek q q ,
where H 2 (p) = -p log 2 p -(1 -p) log 2 (1 -p) is the binary entropy function.

The proof follows from Lemma 6.5.1 and the entropy bound for the sum of binomial coefficients, see, e.g., [START_REF] Galvin | Three tutorial lectures on entropy and counting[END_REF] (Theorem 3.1). Characterization of the existence of MLE for e(B k q ) and the related sharp thresholds seem to be hard for general q, even for q = 2, see Remark 6.5.2. In the next section we discuss the products of k -q Rademacher functions for fixed q ∈ N (q ≤ k). We especially focus on the products of k -1 and k Rademacher functions. Below we characterize the existence of MLE for e(B k k-1 ). As we will see, we get a qualitatively different result than that in Section 6.3.1. Let E and O be the sets of all those points in Q k that have an even and odd number of positive coordinates, respectively. . To this end, we consider the hypercube G Q k , defined as the graph with vertices in Q k and edges between all pairs of points which differ by exactly one coordinate. Thus, For the converse implication, we consider q ∈ {0, . . . , k} and (k -q)-subcubes defined as follows, q i=1 H j i , (6.5.3) where 1 ≤ j 1 < j 2 < . . . < j q ≤ k and H j i = H + j i or H - j i , see (6.3.3). When q = k -1, the intersection, or a 1-cube, is a pair of points in Q k which differ by exactly one coordinate, so they have a different parity. Moreover, each such pair can be obtained in this way. Using (6.5.3), as in the proof of Lemma 6.5.1 we see that 1 {e,o} ∈ B k k-1 for each e ∈ E and o ∈ O. Furthermore,

V (G Q k ) = Q k and E(G Q k ) = {{x, y} ∈ Q k ×Q k : |{j :

Control by oscillations

λ U defined in Section 6.2 may be thought of as a specific measure of oscillation of ϕ. Of course, λ U ≥ 0. Furthermore, for every c ∈ R, λ U (ϕ + c) = λ U (ϕ), ϕ ∈ B, (6.6.1)

and for every (positive number) k > 0 we have (homogeneity), λ U (kϕ) = kλ U (ϕ), ϕ ∈ B, k ≥ 0. (6.6.2)

If U = X , then λ X (-ϕ) = λ X (ϕ) for ϕ ∈ B, and so λ X is a seminorm. Clearly, λ U ≤ λ X . However, if there is a nontrivial ϕ ∈ B + such that ϕ = 0 on U , then λ U (ϕ) = sup X ϕ > 0 but λ U (-ϕ) = 0. The following result is the engine of Theorem 6.2.2. Lemma 6.6.1. U ⊂ X is the set of uniqueness for B + if and only if λ U is comparable with λ X on B, i.e., there exist constants c 1 , c 2 > 0 such that c 1 λ X (ϕ) ≤ λ U (ϕ) ≤ λ X (ϕ) for all ϕ ∈ B.

Proof. We first prove the "if" part. Assume U is not a set of uniqueness for B + . Then there exists a nonzero function ϕ ∈ B + such that ϕ = 0 on U . We have λ U (-ϕ) = 0 and λ X (-ϕ) > 0, hence λ U and λ X are not comparable on B.

We now prove the "only if" part, which is delicate. For all ϑ, ϕ ∈ B we have

λ U (ϑ + ϕ) ≤ max X ϑ + max X ϕ -min U ϑ -min U ϕ = λ U (ϑ) + λ U (ϕ) ≤ λ U (ϑ) + λ X (ϕ).
It follows that λ U (ϑ) ≥ λ U (ϑ -ϕ) -λ X (ϕ), hence λ U (ϑ + ϕ) ≥ λ U (ϑ) -λ X (ϕ).

Therefore, vertλ U (ϑ + ϕ) -λ U (ϑ)vert ≤ λ X (ϕ). As a consequence, λ U is continuous on B. We will prove that there is a number h > 0 such that λ U (ϕ) ≥ hλ X (ϕ) for every ϕ ∈ B. Let S = {ϕ ∈ B : min X ϕ = 0 and max X ϕ = 1}. Let ϕ ∈ S. If λ U (ϕ) = 0, then ϕ = 1 on U . Consider φ = 1 -ϕ. Clearly, φ ≥ 0 and φ = 0 on U . It follows that φ = 0 on X , because U is of uniqueness. Then ϕ ≡ 1, which contradicts the assumption ϕ ∈ S. Therefore, λ U (ϕ) > 0.

Since S is compact and λ U is continuous, h := min S λ U > 0. By (6.6.2) and (6.6.1) we obtain λ U (ϕ) ≥ hλ X (ϕ) for all ϕ ∈ B. a partition of G N . We observe that the maps

G → G \ S 1 , G → G \ S 2 , G → G \ (S 1 ∪ S 2 )
are bijections between G N 10 , G N 02 , G N 12 , respectively, and G N 00 . Also, for every G ∈ G N ,

C(G) = C(G \ S 1 ) = C(G \ S 2 ) = C(G \ (S 1 ∪ S 2 )).
Put differently, C(G) does not depend on the edges (r 1 , s 1 ) and (r 2 , s 2 ). As in the proof of Lemma 6.4.1, we obtain P ((r 1 , s 1 ) , (r 2 , s 2 ) ∈ E (G)) = e cr 1 ,s 1 e cr 2 ,s 2 1 + e cr 1 ,s 1 + e cr 2 ,s 2 + e cr 1 ,s 1 e cr : I q ⊂ {0, . . . , k} and |I q | ≤ q    .

We have B = B k q , because r 0 = 1 Q k , r i = 21 H + i -1 Q k and by induction it is easy to see that for every S ⊂ {1, . . . , k} and |S| < q, if Walsh function w S ∈ B then their product with Rademacher function w S r i ∈ B, for any i = 0, . . . , n. Note that for any permutation σ of {1, 2, . . . , q},

1 H + i 1 1 H + i 2 • • • 1 H + iq = 1 H + i σ(1) 1 H + i σ(2) • • • 1 H + i σ(q) .
The functions 1 Q k and 1

H + i 1 • • • 1 H + iq
, 1 ≤ i 1 ≤ . . . ≤ i q ≤ k, are linearly independent. Indeed, assume that r := α 0 1 Q k + i 1 ,...,iq∈{1,...,k}

α i 1 •••iq 1 H + i 1 • • • 1 H + iq = 0.
There are points x 0 ∈ k i=1 H - i , x i 1 . . . x iq ∈ l∈{i 1 ,...,iq} H - l ∩ l̸ =i 1 ,...,iq H - l for each 1 ≤ i 1 ≤ i 2 ≤ . . . ≤ i q ≤ k. We obtain α 0 = r(x 0 ) = 0 and α i 1 •••iq = r(x i 1 •••iq ) = 0 as needed.

Propagation of extrema, relative interior and the criterion of Barndorff-Nielsen

In this section we give auxiliary results, but also explain connections to the criterion of Barndorff-Nielsen. Let B be an arbitrary linear subspace of R X . In Corollary 6.6.1 below we adapt the criterion in Theorem 6.2.2 to such e(B). Let B ′ be the linear space spanned by B and 1.

Lemma 6.6.2. If U ⊂ X , then ϕ = min X ϕ on U implies ϕ = min X ϕ on X for every ϕ ∈ B if and only if ϕ = max X ϕ on U implies ϕ = max X ϕ on X for every ϕ ∈ B.

To close the circle of ideas, we give a self-contained proof of Theorem 6.6.1, which may also be used to obtain Theorem 6.2.2 from Theorem 6.6.1.

Proof of Theorem 6.6.1. By the discussion in this section we know very well that MLE for x 1 , ..., x n and e(B) exists if and only if for every ϕ ∈ B, min X ϕ < max X ϕ implies min X ϕ < For clarity, we recall that we agreed in Example 6.1.2 that the existence of MLE for x 1 , ..., x n ∈ X and e(B) is the same as the existence of MLE for x 1 , ..., x n and the exponential family given by the canonical statistics T and (6.1.10), and that it is equivalent to the existence of MLE for the sample y 1 := T (x 1 ), ..., y n = T (x n ) ∈ R d and the standard exponential family in (6.1.11). From the above discussion we also see that the convex hull C and the notion of relative interior are merely auxiliary objects to express the property in Lemma 6.6.4, or the propagation of extrema property.

Chapter 7

On Laplacian of Graphical Models in Various Graphs

Introduction

The content of this chapter comes from the published article [START_REF] Skalski | Remarks on Laplacian of graphical models in various graphs[END_REF] of the author of the dissertation from the conference "Geometric Science of Information 2021" in Sorbonne University, Paris. Let G = (V, E, C) be a simple undirected graph, where V = {1, 2, . . . , n} is a set of vertices, E ⊂ n 2 is a set of edges and C ⊂ V is a set of source vertices, which will be called later as a "root set" or a "root". As a degree deg(v) of a vertex v ∈ V we treat a number of its neighbours. In our convention the graph Laplacian is defined as L(G) = {l i,j } 1≤i,j≤n with l i,i = deg(i), l i,j = -1 if {i, j} is an edge in G and zeros otherwise, cf. e.g. [START_REF] Brouwer | Spectra of graphs[END_REF]. Note that the graph Laplacian is a singular matrix, since its entries in each row (and each column) sum up to 0. Therefore we introduce an augmented graph Laplacian by adding 1 to every entry l with E C being a square matrix with 1 in (c, c) for c ∈ C and zeros everywhere else. We define L * c (G) = {l * i,j } 1≤i,j≤n with l * i,j = l i,j + 1 {i=j∈C} .

Trees

Let T = (V, E) be an undirected tree. We may orient it in a following way: Choose one root vertex C = {c}. Then we orient every edge in a direction from c. Following this method we may induce a partial order ≤ on the set of vertices such that v ≤ v ′ if and only if there exists a directed path from v ′ to v. For every vertex v ∈ V we define its ancestry AN(v) := {w ∈ V : v ≤ w} as a set of vertices in a unique path from v to c. Note that both the partial order and AN depend strictly on the choice of c. Now consider an n-dimensional Gaussian random variable (X 1 , . . . , X n ) with a covariance matrix Σ = {σ i,j } 1≤i,j≤n such that σ i,j = |AN(i) ∩ AN(j)| for every 1 ≤ i, j ≤ n. Analogously, if j ∈ AN(i), then σ k,j = σ i,j for every k ∼ i. Therefore:

(L * c Σ) i,j = n k=1 l * i,k σ k,j = n k=1 l i,k • σ k,j = deg(i) • σ i,j -deg(i) • σ i,j = 0.
The same argument may be applied in the only case left, when i ̸ = j; i, j ̸ = c, i / ∈ AN(j) and j / ∈ AN(i). Therefore here also (L * c Σ) i,j = 0.

Daisy graphs

The daisy graphs (cf. e.g. Nakashima and Graczyk [START_REF] Nakashima | Wigner and wishart ensembles for graphical models[END_REF]) may be understood as a notion between a complete bipartite graph and a complete graph. To be more specific, the daisy graph D a,b is built as a sum of the complete bipartite graph K a,b and a complete subgraph K a , i.e.

D a,b := (V = V A ⊔ V B , E), V = V A ⊔ V B , (x, y) ∈ E ⇐⇒ {x, y} ∩ V A ̸ = ∅.
We may interpret V A and B B as the internal and the external part of a daisy, respectively.

To give an intuition, graphical models based on daisy graphs may be useful for analysis of a internal features of data (without knowledge of any independence among them) and b mutually conditionally independent external factors, which can influence the internal environment. Note that for b = 1 we have D a,1 being a complete graph K a+1 and for a = 1 we have D 1,a being a star graph, which is a tree (cf. 7.2). Without loss of generality we imply such ordering on vertices that the internal vertices precede the external ones. Moreover, concerning cases of one rooted internal (external) vertex we label it as the first (last) vertex. The exact formulas for (L * ) -1 (D a,b ) are presented below:

Wiener process (W t ) t≥0 is Cov(W s , W t ) = min{s, t}. For example, (W 1 , W 2 , . . . , W n ) has its covariance matrix equal to (Σ) i,j = min{i, j}. At Theorem 7.2.1 we proved that this is exactly the inverse of the augmented Laplacian of a path graph with an initial vertex in one of its endpoints. This observation stays consistent with the conditional independence of W t 1 and c = 1 W t 2 under W t for any t 1 < t < t 2 . Similarly, the broader class of trees and their augmented Laplacians can be connected with sums of the standard Gaussian random variables 'branched' according to the underlying tree graph. Therefore we may consider the model corresponding to the cycle graph as the Wiener model (W 1 , . . . , W n , W n+1 ) conditioned by W n+1 = W 1 . This gives a sum of a random variable W 1 ∼ N (0, 1) and a Brownian bridge "tied down" at 1 and (n + 1). Therefore the covariance matrix of (W 1 , . . . , W n ) is equal to σ i,j = 1 + (i -1)(n -j + 1) n , i ≤ j, cf. Theorem 7.3.1.

Z 1 + Z 2 Z 1 + Z 2 + Z 3 Z 1 + Z 2 + Z 4 Z 1 + Z 2 + Z 5 (
In order to find a model with a covariance matrix equal to

[L * 1 (K n )] -1 i,j =        1 if i = 1 or j = 1, 1 + 2 n if 1 < i = j, 1 + 1 n else,
observe that all vertices (except of the initial one) are isomorphic and connected, therefore each of their correspondent random variables are mutually equally dependent Let c = 1 and let Z 1 , . . . , Z n be i.i.d. random variables from N (0, 1). Therefore X 1 , X 2 , . . . , X n are of the form

                 X 1 = Z 1 X 2 = Z 1 + αZ 2 + β(Z 3 + Z 4 + . . . + Z n ) X 3 = Z 1 + αZ 3 + β(Z 2 + Z 4 + . . . + Z n ) • • • X n = Z 1 + αZ n + β(Z 2 + Z 3 + . . . + Z n-1 ).
The restriction on the covariance matrix of (X 1 , . . . , X n ) induces a system of equations, which is satisfied only for β = Le dernier des applications relie la théorie des schémas graphiques en statistique avec la notion de matrices laplaciennes de graphes et de processus de Wiener discrétisés.

Title: Geometric and Combinatorial Aspects of Statistical Models.

Keywords: linear regression, SLOPE, polyhedron, MLE, random graph

Abstract: We concern new applications of discrete geometry and combinatorics in modern statistics. First of them focuses on the use of penalized linear regresion methods. We start our discussion with the SLOPE estimator. We examine the notion of the SLOPE pattern, which maintains the information about the support, sign and ranking between the regression coefficients. We provide the conditions, under which SLOPE recovers its pattern when the design matrix is orthogonal. For a general class of fixed design matrices we provide the SLOPE irrepresentability condition, which is necessary and sufficient for the pattern recovery in the noiseless case, and we illustrate it geometrically. Later on, we consider the asymptotic case. We also study the wider class of penalized estimators, called the polyhedral gauges. Second application concerns the existence of the MLE estimator for discrete exponential families. We give its new characterization based on the notion of the set of uniqueness. Later on, we inspect the size of independent identically distributed samples which is needed to ensure its existence with high probability. For that reason we use the notions from the analysis of discrete hypercubes and apply our results in the environment of random graphs. Last application connects the theory of graphical models in statistics with the notion of graph Laplacian matrices and discretized Wiener processes.

  . Estymator OLS jest zdefiniowany jako wektor b minimalizujący odległość euklidesową między Y , a Xb: βOLS := arg min b ∈R p ∥Y -Xb∥ 2 . xiii xiv CHAPTER 0. WSTĘP (INTRODUCTION IN POLISH)

Definicja PL. 2 . 1 (Fakt PL. 2 . 1 (

 2121 Wzorzec SLOPE). Wzorzec SLOPE patt : R p → Z p jest zdefiniowany następująco: patt(b) i = sign(b i )rank(|b i |), gdzie rank(|b i |) ∈ {1, 2, . . . , k} jest zdefiniowany jako liczba takich |c j |, dla których |b i | ≥ |c j |, gdzie |c 1 |, |c 2 |, . . . , |c k |, k ≤ p są niezerowymi różnymi wartościami spośród |b 1 |, . . . , |b p |. Przyjmujemy rank(0) = 0. Wzorzec SLOPE zachowuje informację nie tylko o znaku wektora, ale również o jego klastrach, tzn. zbiorach współrzędnych mających tę samą wartość bezwzględną oraz o hierarchii między tymi wartościami. xvi CHAPTER 0. WSTĘP (INTRODUCTION IN POLISH) Własności wzorca SLOPE [156]). (a) Dla każdego 1 ≤ l ≤ ∥patt(b)∥ ∞ istnieje takie j, że |patt(b) j | = l, (b) sign(patt(b)) = sign(b), (c) |b i | = |b j | ⇒ |patt(b) i | = |patt(b) j |, (d) |b i | > |b j | ⇒ |patt(b) i | > |patt(b) j |. Example PL.2.2. patt((4, 0, -1.5, 1.5, -4) ′ ) = (2, 0, -1, 1, -2) ′ . Mówimy, że β SLOP E odtwarza wzorzec β, kiedy patt β SLOPE = patt(β).

  x∈X e ϕ(x) µ(x), ψ(ϕ) = log Z(ϕ), a także gęstość wykładniczą p = e(ϕ) = e ϕ-ψ(ϕ) = e ϕ /Z(ϕ). PL.4. MODELE GRAFICZNE xix Rodzina wykładnicza rozpięta przez B jest ukazana poniżej. e(B) := {p = e(ϕ) : ϕ ∈ B}.

  xx CHAPTER 0. WSTĘP (INTRODUCTION IN POLISH) klasycznego procesu Wienera.

  βLASSO , • Λ est une suite arithmétique ⇒ βSLOPE = βOSCAR . Pour une description plus détaillée d'histoire des recherches sur l'estimateur SLOPE, voir la Section 4.1.1. Dans notre recherche, nous nous concentrons sur une autre propriété importante de SLOPE, à savoir la récouvrement des caractéristiques importantes d'un vecteur de coefficients de régression β, appelé son SLOPE schéma. Il s'agit d'un analogue du vecteur de signe dans LASSO et, comme son prédécesseur, il peut être entièrement décrit par le sous-différentiel de la norme pénalisante. Pour être plus spécifique, laissez k être le nombre de clusters de patt(β) = (m 1 , . . . , m p ) ′ c'està-dire le nombre de composantes distinctes non nulles de |β|. Définition FR.2.1 (SLOPE schéma). Le SLOPE schéma est une fonction patt : R p → Z p telle que patt(b) i = sign(b i )rank(|b i |), où rank(|b i |) ∈ {1, 2, . . . , k} est défini comme le nombre de |c j |'s satisfaisant |b i | ≥ |c j |, où |c 1 |, |c 2 |, . . . , |c k |, k ≤ p, sont des valeurs distinctes non nulles parmi |b 1 |, . . . , |b p |. Nous adoptons la convention que rank(0) = 0.

FR. 4 .

 4 MODÈLES GRAPHIQUES xxix Pour une fonction à valeur réelle ϕ, nous définissons les fonctions de partition et de log-partition, Z(ϕ) = x∈X e ϕ(x) µ(x), ψ(ϕ) = log Z(ϕ), respectivement, et densité exponentielle p = e(ϕ) = e ϕ-ψ(ϕ) = e ϕ /Z(ϕ). La famille exponentielle couverte par l'ensemble B est e(B) := {p = e(ϕ) : ϕ ∈ B}.

Definition 1 . 2 . 1 (

 121 SLOPE pattern). The SLOPE pattern is a functionpatt : R p → Z p such that patt(b) i = sign(b i )rank(|b i |),where rank(|b i |) ∈ {1, 2, . . . , k} is defined to be the number of|c j |'s satisfying |b i | ≥ |c j |, where |c 1 |, |c 2 |, . . . , |c k |, k ≤ p, are distinct nonzero values among |b 1 |, . . . , |b p |.We adopt the convention that rank(0) = 0.

Fact 1 . 2 . 1 (

 121 Basic properties of SLOPE pattern[START_REF] Schneider | The geometry of uniqueness, sparsity and clustering in penalized estimation[END_REF]). (a) for every 1 ≤ l ≤ ∥patt(b)∥ ∞ there exists j such that |patt(b) j | = l, (b) sign(patt(b)) = sign(b) (sign preservation), (c)

Figure 1 . 1 :

 11 Figure 1.1: βSLOPE and βOLS in orthogonal design: X ′ X = I p for Λ = (2, 1) ′ .

Figure 1 . 2 :

 12 Figure 1.2: βSLOPE and βOLS for X ′ X = 1 0.6 0.6 1 and Λ = (2, 1) ′ .

Remark 2 . 3 . 1 .Definition 2 . 3 . 3 (Definition 2 . 3 . 4 (

 231233234 [START_REF] Boyd | Convex Optimization[END_REF] Sec. 2.1.4] aff(C) is the smallest affine set containing C. Relative interior).[START_REF] Boyd | Convex Optimization[END_REF] Sec. 2.1.3] Let C ⊂ R p . The relative interior of a set C is defined as the interior of C with respect to its affine hull aff(C): ri(C) := {x ∈ C : B ∥•∥ (x, r) ∩ aff(C) ⊂ C for some r > 0}. Relative boundary).[START_REF] Boyd | Convex Optimization[END_REF] Sec. 2.1.3]

Remark 2 . 3 . 4 .Definition 2 . 3 . 10 ( 2 Definition 2 . 3 . 11 (Definition 2 . 3 . 13 .

 2342310223112313 [101, Prop. III. 1.2.1] A non-empty closed and convex set C is compact if and only if rec(C) = {0}. Orthogonal projection). [33, Sec. 8.1] Let C ⊂ R p be a closed set and let x 0 ∈ R p . The orthogonal projection of x 0 onto C is defined as P C (x 0 ) := arg min x ∈ C ∥x -x 0 ∥ Half-space). [34, p. 9] Let y ∈ R p and α ∈ R. Then K(y, α) := {x ∈ R p : ⟨x, y⟩ ≤ α}. Definition 2.3.12 (Hyperplane). [34, p. 9] Let y ∈ R p and α ∈ R. Then H(y, α) := {x ∈ R p : ⟨x, y⟩ = α}. CHAPTER 2. PRELIMINARIES Let A ⊂ R p be an affine space, i.e.

Definition 2 . 3 . 15 (

 2315 Pattern equivalence class).

  Moreover, as ⟨y, v⟩ ≤ ⟨y, x⟩ for every v ∈ B * , we get y ∈ ⟨y, x⟩B. Now it suffices to prove that for every v ∈ F we have ⟨y, v⟩ = ⟨y, x⟩. Take any v ∈ F . Since x ∈ ri(F ) ⊂ F , we have N B * (x) ⊂ N B * (v). Then ⟨y, w -x⟩ ≤ 0 for every w ∈ B * , which implies that ⟨y, v -x⟩ ≤ 0, i.e. ⟨y, v⟩ ≤ ⟨y, x⟩. On the other hand, y ∈ N B * (v). Therefore ⟨y, w -v⟩ ≤ 0 for every w ∈ B * , which implies that ⟨y, x -v⟩ ≤ 0, i.e. ⟨y, x⟩ ≤ ⟨y, v⟩. Thus for every v ∈ F we have ⟨y, v⟩ = ⟨y, x⟩, hence y ∈ ⟨y, x⟩F o ⊂ R + F o .

Definition 2 . 3 . 23 (

 2323 Thresholded penalized least squares estimator).

Theorem 2 . 3 . 11 (

 2311 Sion).[START_REF] Aubin | Mathematical methods of game and economic theory[END_REF] Ch. 7, Theorem 7.] Suppose that (a) X and Y are convex compact subsets, (b) for all y ∈ Y the function x → f (x, y) is quasiconvex and lower semi-continuous, (c) for all x ∈ X the function y → f (x, y) is quasiconcave and upper semi-continuous, Then there exists a saddle point {x, y}.

2 . 1 )

 21 Proposition 3.2.1. Let C Λ be the unit ball in the dual SLOPE norm. Then, for each b ∈ R p ,

  Since for each fixed π ∈ C Λ , the function r(b, π) is convex with respect to b ∈ M, any point b π ∈ M, at which the gradient ∂r(b, π) ∂b is zero, is a global minimum. If we rewrite r(b, π) as

Figure 3 . 1 :

 31 Figure 3.1: The dual unit ball C Λ/n for Λ = (200, 100) ′ and examples of βSLOPE and βOLS in the orthogonal design for n = 50 and p = 2. The labels of each colored set refer to the pattern of βSLOPE for βOLS lying in this set. The arrows point from ( βOLS -βSLOPE ) to βOLS .

Proposition 3 . 4 . 1 .

 341 Consider the linear regression model (3.4.1).

  we get the assertion by Lemma 3.4.1. Proof of Theorem 3.4.1. (a) It follows from Theorem 3.2.1 that there exists a vector π

Remark 3 . 4 . 2 .

 342 which provides a contradiction. This proves the first part of Theorem 3.4.1 (b). To prove the second part, note that strong consistency requires convergence for any value of the parameter β. The proof of Theorem 3.4.1 (b) does not exclude the possibility that β SLOPE n a.s.

.4. 5 )

 5 Then we have patt( βSLOPE n ) a.s.

.4. 8 )

 8 Take any j ∈ S i . Since both β OLS j (n) and β OLS i (n) have the normal distribution with the same mean, by Lemma 3.4.1, we have lim n)) 1/2+δ = 0 a.s. In view of (3.4.8) and (3.4.5), this implies that (3.4.7) holds true for large n. Hence, (a) follows. It remains to establish (c). Assume that β p 0 > 0 = β p 0 +1 = . . . = β p . Clearly, condition (a) from Theorem 3.3.2 is satisfied thanks to the strong consistency of the OLS estimator. For (b), we have for k = 1, 2, . . . , p 0 , 1 n

  X i,(2 * j-1) = sin(2πij/n) and X i,(2 * j) = cos(2πij/n) for i = 1, . . . , 100 and j = 1, . . . , 150. Here β ∈ R p is a vector consisting of two clusters: 20 coordinates with absolute value 100 and 20 coordinates with absolute value 80. The absolute values of coordinates of β are sorted in a decreasing way. The signs of the nonzero coordinates are chosen independently with random uniform distribution. To avoid large bias caused by the shrinkage nature of LASSO and SLOPE, we debias them by combining with the OLS method. For that reason we use the pattern matrix U M and the clustered design matrix X M , which is based on the SLOPE pattern. To perform the debiased SLOPE, we begin with recovering the support and clusters of a true vector β with SLOPE. Then, using the obtained SLOPE pattern M , we replace the design matrix with its clustered version X M = XU M . Then we perform the Ordinary Least Squares regression for the model Y = X M b + ε, where b consists only of distinct absolute values of βSLOPE .

Figure 3 . 2 :

 32 Figure 3.2: Comparison of signal denoising by OLS (a), LASSO (b), debiased LASSO (c) and debiased SLOPE (d) on the coordinates [120, 125] of the regression model Y = Xβ + ε. The black lines correspond to the true values of Xβ. The red lines correspond to the estimators Y = X β.

Figure 3 . 3 :

 33 Figure 3.3: Signal denoising by debiased SLOPE on all coordinates of the regression model Y = Xβ + ε. The (almost overlapping) black line and the red line correspond respectively to the true values of Xβ and to Y = X βSLOPE .

Figure 3 . 4 :

 34 Figure 3.4: Pattern recovery by debiased LASSO (left) and by debiased SLOPE (right) in the same setting as above.

  π∈C Λ r(β * , π) = r(β * , π * ). Moreover, from the definition of the point β * it is seen that r(β * , π * ) = min β∈M r(β, π * ). These two facts imply that min β∈M max π∈C Λ r(β, π) ≤ max π∈C Λ r(β * , π) = r(β * , π * ) = min β∈M r(β, π * ) ≤ max π∈C Λ min β∈M r(β, π). Since max π∈C Λ min β∈M r(β, π) ≤ min β∈M max π∈C Λ r(β, π) (by the max-min inequality), we have the equality throughout. This completes the proof. Proof of Lemma 3.3.1. Observe that

Figure 4 . 1 :

 41 Figure 4.1: Comparison of LASSO and SLOPE when the cluster structure is present in the data.Here n = 100, p = 200, the rows of the matrix X are simulated as independent binary Markov chains, with the transition probability 0.0476 (corresponding to the genetic distance of 5 centimorgans). The correlation between i th and the j th column of X decays exponentially as 0.9048 |i-j| . The first k = 30 columns of X are associated with Y and their nonzero regression coefficients are all equal to 40 (other details are provided in the text). The left panel represents the value of the tuning parameter for LASSO (solid line) and the sequence of tuning parameters for SLOPE (crosses). The sequences are selected such that both LASSO and SLOPE recover their corresponding patterns with a minimal bias. The right panel represents the LASSO and SLOPE estimates.

Definition 4 . 2 . 1 .Example 4 . 2 . 1 .

 421421 The SLOPE pattern is a function patt : R p → Z p such that patt(b) i = sign(b i )rank(|b i |), where rank(|b i |) ∈ {1, 2, . . . , k} is defined to be the number of |c j |'s satisfying |b i | ≥ |c j |, where |c 1 |, |c 2 |, . . . , |c k |, k ≤ p, are distinct non-zero values among |b 1 |, . . . , |b p |. We adopt the convention that rank(0) = 0. We also recall that R k+ = {κ ∈ R k : κ 1 > . . . > κ k > 0} and that M p is the set of SLOPE patterns of b ∈ R p . Definition 2.2.4 implies that for 0 ̸ = M ∈ M SLOPE p and k = ∥M ∥ ∞ , for b ∈ R p we have patt(b) = M ⇐⇒ there exists κ ∈ R k+ such that b = U M κ. Let b = (2, 5, -2, 0, 8) ′ . Then M = patt(b) = (1, 2, -1, 0, 3) ′ and for κ = (8, 5, 2) ′ we have

Example 4 . 2 . 2 .

 422 then the pattern M = (M 1 , . . . , M p ) ′ leads naturally to reduce the dimension of the design matrix X in the regression problem, by replacing X by X M . Actually, if patt(β) = M , then Xβ = XU M κ = X M κ for κ ∈ R k+ . In particular, (a) null components m i = 0 lead to the elimination of columns X i from the design matrix X, (b) a cluster K ⊂ {1, . . . , p} of M (a subset of M having coordinates equal in absolute value) leads to replace the columns (X i ) i∈K by one column equal to their signed sum: i∈K sign(m i )X i . Let b be from Example 4.2.1, X = (X 1 |X 2 |X 3 |X 4 |X 5 ) and Λ =

  recall the subdifferential of a norm ∥ • ∥ at b (see e.g. [101, Def. VI.1.2.1]:∂∥ • ∥(b) = {v ∈ R p : ∥z∥ ≥ ∥b∥ + v ′ (zb) ∀ z ∈ R p }, = {v ∈ R p : ∥v∥ * ≤ 1 and v ′ b = ∥b∥}.

( 4 . 2 . 1 )Proposition 4 . 2 . 1 .

 421421 For the sorted ℓ 1 norm, geometrical descriptions of the subdifferential at b ∈ R p have been given in the particular case where b 1 ≥ . . . ≥ b p ≥ 0 [61, 156, 175]. Below, for an arbitrary b ∈ R p , we propose a new and useful formula for the subdifferential of the sorted ℓ 1 norm. This representation is the crux of the mathematical content of the present chapter. Let b ∈ R p and M = patt(b). Then:

Proposition 4 . 2 . 2 .

 422 Let a, b ∈ R p and Λ = (λ 1 , . . . , λ p ) ′ where λ 1 > . . . > λ p > 0. We have patt(a) = patt(b) if and only if ∂J Λ (a) = ∂J Λ (b).

Remark 4 . 3 . 3 .

 433 Let 0 ̸ = S ∈ {-1, 0, 1} p and k = ∥S∥ 1 (k is the number of nonzero components of S). The signed matrix U S ∈ R p×k is defined by U S = (diag(S)) supp(S) , where diag(S) ∈ R p×p is a diagonal matrix and (diag(S)) supp(S) denotes the submatrix of diag(S) obtained by keeping columns corresponding to indices in supp(S). Observe that for any 0 ̸ = β ∈ R p there exists a unique S ∈ {-1, 0, 1} p and a unique κ 0 ∈ (0, ∞) k such that β = U S κ 0 . Define the reduced matrix X S and the reduced parameter λ S by X S = XU S and λ S = λ1 k , where

Definition 4 . 3 . 1 . 2 ) 4 . 3 . 1 .

 4312431 Let X ∈ R n×p , Λ ∈ R p+ and 0 ̸ = β ∈ R p . The noiseless pattern recovery by SLOPE is defined as∃ α > 0 ∃ β ∈ S X,αΛ (Xβ) such that patt( β) = patt(β). (4.3.Corollary Let X ∈ R n×p , Λ ∈ R p+ and 0 ̸ = β ∈ R p . The noiseless pattern recovery by SLOPE is equivalent to

Figure 4 . 2 :

 42 Figure 4.2: SLOPE solution paths for p = 2 and Λ = (4, 2) ′ in the noiseless case. β is equal respectively to (5, 0) ′ (left panel) and (5, 3) ′ (right panel). As may be seen, the pattern of β = (5, 0) ′ cannot be recovered in the noiseless case for any scaling of Λ. The pattern of β = (5, 3) ′ is recovered for α ∈ (0, 0.4).

Proposition 4 . 4 . 2 (Figure 4 . 4 :Figure 4 . 5 : 1 )Figure 4 . 6 :

 4424445146 Figure 4.4: Intersection Π of col(X ′ X M ) = col((1, 1) ′ ) and aff(∂J Λ (M )) for X ′ X = 1 2/3 2/3 1 , Λ = (4, 1) ′ and patt(β) = M = (1, 1) ′ . Since Π = (2.5, 2.5) ′ ∈ ∂J Λ (M ), then in the noiseless case, by Proposition 4.4.1, SLOPE recovers the pattern M of β.

Figure 4 . 7 :

 47 Figure 4.7: Comparison of βSLOP E , βOLS and their difference (X′ X) -1 Xε ∈ (X ′ X) -1 C Λ for

Figure 4 . 8 :

 48 Figure 4.8: C Λ in R p for p = 3 with its two-dimensional subset corresponding to the intersection of C Λ with row(X) from [156, Example 4]. One may observe that in this example there are only 17 accessible patterns out of all 147 patterns for p = 3.

  n -P M )Xβ = (I n -P M ) X M s = 0, where β = U M s and s ∈ R ∥M∥∞+ . By Theorem 4.3.1, the probability of the pattern recovery by SLOPE is upper bounded by P ∃ β ∈ S X,αΛ (Y ) such that patt( β) = patt(β) ≤ P (J * Λ (π α ) ≤ 1) , 0 if Λ M / ∈ col( X ′ M ).

( 4 . 5 . 2 )Definition 4 . 5 . 1 (

 452451 The first point in Theorem 4.5.1 shows that the probability of pattern recovery matches with the upper bound (4.5.2) when gaps between different absolute values of terms of β are large enough. The last point provides the pattern consistency by SLOPE. Before stating this Theorem, we introduce the open SLOPE irrepresentability condition. Open SLOPE irrepresentability condition).

Figure 4 . 9 . 4 . 5 . 1 .

 49451 Theorem Let X ∈ R n×p , 0 ̸ = M ∈ M SLOPEp , and Λ = (λ 1 , . . . , λ p ) ′ ∈ R p+ . Consider a sequence of signals (β (r) ) r≥1 with pattern M :

  Thus, if β does not have any non-trivial clusters nor zeros, i.e. ∥patt(β)∥ ∞ = p, then α n /n → 0 suffices for patt( βSLOPE n ) a.s.

Theorem 4 . 5 . 3 .

 453 Under the assumptions A, B1, C, the following statements hold true. (a) If B2 is additionally satisfied and moreover α n = √ n, then

1+c 2 →

 2 ∞, then under the strong LASSO irrepresentability condition, one has sign( βLASSO n ) a.s.

  ε n = (ε 1 , . . . , ε n ) ′ , where (ε i ) i are independent random variables such that E[ε n ] = 0 and Var(ε n ) = σ 2 for all n, and sup n E[|ε n | r ] < ∞ (4.6.1)for some r > 2.

Theorem 4 . 6 . 1 .If ( 4 . 5 . 5 )

 461455 Assume A', B" and C. Suppose that (α n ) n satisfies lim is satisfied, then patt( βSLOPE n ) a.s.

Figure 4 . 10 :

 410 Figure 4.10: The estimates of probability of pattern recovery by SLOPE as a function of c where c = β 1 = . . . = β 100 > 0. For each point the probability is computed via 10 5 Monte-Carlo experiments. The scaling parameter α 0.95 ≈ 1.391 is chosen to fix the upper bound at 0.95. Note that when c is large, the probability of pattern recovery is approximately equal to 0.95.

Figure 4 .

 4 11, X ∈ R n×100 has i.i.d. N(0, 1) entries, β 1 = . . . = β 100 = 1 and α 0.95 ≈ 1.391.Actually, since n -1 X ′ X converges to I 100 , when patt(β) = (1, . . . , 1) ′ the Gaussian vector involved in the limiting probability has the same mean and covariance as (4.7.1).

Figure 4 . 11 :

 411 Figure 4.11: The estimates of probability of pattern recovery by SLOPE as a function of n. For each point the probability is computed via 10 3 Monte-Carlo experiments.

Figure 4 . 12 :

 412 Figure 4.12: Comparison of LASSO and SLOPE when the cluster structure is present in the data. Here n = 100, p = 200, and the correlation between i th and j th column of X is equal to 0.9048 |i-j| . First k = 100 columns of X are associated with Y and their nonzero regression coefficient are all equal to 40. The SLOPE and LASSO irrepresentability conditions are not satisfied, but SLOPE, contrary to LASSO, satisfies the accessibility condition.

  Consequently, Λ /∈ ∂J Λ (b), which again leads to a contradiction.Let ψ be an orthogonal transformation defined by ψ :x ∈ R p → (v 1 b r(1) , . . . , v p b r(p) ) where v 1 , . . . , v p ∈ {-1,1} and let r be a permutation on {1, . . . , p}. Before proving Proposition 4.2.2, let us recall that for any a, b ∈ R p we have J Λ (b) = J Λ (ψ(b)), J * Λ (b) = J * Λ (ψ(b)) and b ′ a = ψ(b) ′ ψ(a), therefore ∂J Λ (ψ(b)) = ψ(∂J Λ (b)). Proof of Proposition 4.2.2. If patt(a) = patt(b), then, according to Proposition 4.2.1, ∂J Λ (a) = ∂J Λ (b). Set M = patt(a) and M = patt(b). It remains to prove that if ∂J Λ (a) = ∂J Λ (b), then M = M . Since the subdifferential ∂J Λ (a) depends on a only through its pattern, then by Proposition 4.2.1 we have ∂J Λ (a) = ∂J Λ (M ) and similarly ∂J Λ (b) = ∂J Λ ( M ). First let us assume that M

4 . 9 . 3 1

 4931 c) The equalities hold in (4.9.2) for j ∈ {n 1 , n 2 , . . . , n k }, where n j = |{i :|m i | ≥ k + 1 -j}| with (M 1 , . . . , M p ) ′ =patt(b). Proof of Theorem 4.3.Proof of Theorem 4.3.1. Necessity. Let us assume that there exists β ∈ S X,Λ (Y ) with patt( β) = M . Consequently, β = U M s for some s ∈ R k+ .

  then, by Theorem 4.3.1, it remains to show that the positivity condition occurs for α > 0 small enough. Since β = U M s for some s ∈ R k+ , where

  k+ and thus, the positivity condition is proven.

4. 9 . 5 1 Lemma 4 . 9 . 2 .

 951492 Proof of Theorem 4.5.Let 0 ̸ = b ∈ R p and M = patt(b). Then the smallest affine space containing ∂J Λ (b) is aff(∂J Λ (b)) = {v ∈ R p : U ′ M v = Λ M }. Proof. According to Proposition 4.2.1 we have aff

Lemma 4 . 9 . 4 .

 494 Let M = patt(β). Assume α n /n → 0. (a) Under A, B1 and C, the positivity condition is satisfied for large n with high probability. (b) Under A, B' and C, the positivity condition is almost surely satisfied for large n.

  For instance, when D is a matrix such that Db = (b 2 -b 1 , . . . , b p -b p-1 ) ′ (denoted D tv below), then the penalty term ∥D.∥ 1 promotes equality between neighbouring components of β. Moreover, this estimator can recover the jump set: {i ∈ [p -1] :

  while both summands are not larger than zero. Thus, ⟨d, x -s⟩ = ⟨d, x -(x -(xs))⟩ = 0. It means that for every s ∈ aff(F ) we have ⟨d, s⟩ = ⟨d, x⟩. It implies that for every s ∈ ----→ aff(F ) we have ⟨d, s⟩ = 0, thus d ∈ ----→ aff(F ) ⊥ . By the arbitrariness of d, we obtain N F (x) ⊂ ----→ aff(F ) ⊥ and then lin(N P

Definition 2 . 3 . 15 (Lemma 5 . 2 . 1 .

 2315521 Pattern equivalence class). [Pattern equivalence class] Let x ∈ R p and let pen be a polyhedral gauge. The pattern equivalence class C x is the set of all vectors having the same subdifferential as x: C x := {w ∈ R p : ∂pen(w) = ∂pen(x)}. Let pen be a polyhedral gauge and x ̸ = 0. Then C x ⊂ N (F x ).

Figure 5 . 2 :

 52 Figure 5.2: Pattern equivalence classes for ℓ ∞ norm, orthogonal X and p = 2: pen(b) = ∥b∥ ∞ = max{|b 1 |, |b 2 |}. On the left the blue polytope is B * . Red and green (unbounded) sets are the preimages, with respect to the projection onto B * , of its vertices and edges. The picture on the right presents pattern equivalence classes of vectors in R 2 .

  ′ . Figure 5.3 illustrates the comparison between projections onto B * and the patterns of β SLOPE for p = 2 and orthogonal X.

Figure 5 . 3 :

 53 Figure 5.3: Pattern equivalence classes for SLOPE in orthogonal design with p = 2 and Λ = (2, 1) ′ : pen(b) = J Λ (b) = λ 1 |b| (1) + λ 2 |b| (2) . On the left the blue polytope is B * . Red and green (unbounded) sets are the preimages, with respect to the projection onto B * , of its vertices and edges. The picture on the right presents pattern equivalence classes of vectors in R 2 .

Figure 5 .Figure 5 . 4 :Figure 5 . 5 :

 55455 Figure 5.4: Pattern equivalence classes for the total variation in orthogonal design with p = 2: pen(b) = |b 1 -b 2 |. On the left the blue polytope is B * . Red and green (unbounded) sets are the preimages, with respect to the projection onto B * , of its vertices and edges. The picture on the right presents pattern equivalence classes of vectors in R 2 .

Proposition 5 . 4 . 1 (

 541 Characterization of accessible patterns). Let X ∈ R n×p and pen : R p → R be a real-valued polyhedral gauge. (a) Geometric characterization: The pattern of β ∈ R p is accessible with respect to X and 5.4. PATTERN RECOVERY IN PENALIZED ESTIMATION λpen if and only if row(X) ∩ ∂pen(β) ̸ = ∅. (b) Analytic characterization: The pattern of β ∈ R p is accessible with respect to X and λpen if and only if for any b ∈ R p we have Xβ = Xb =⇒ pen(β) ≤ pen(b).

Definition 5 . 4 . 2 (Remark 5 . 4 . 1 .Example 5 . 4 . 2 . 1 ,

 5425415421 Noiseless recovery condition). Let pen be a polyhedral gauge, X ∈ R n×p and β ∈ R p . We say that the pattern of β satisfies the noiseless recovery condition with respect toX and pen if ∃λ > 0, ∃ β ∈ S X,λpen (Xβ) such that β pen ∼ β.This condition generalizes the noiseless pattern recovery condition for SLOPE (4.3.2). β = 0 satisfies the noiseless recovery condition with respect to X and pen, because Xβ = 0 and 0 ∈ S X,λpen (0). In other words, the noiseless recovery condition means that in the noiseless case when Y = Xβ, in the solution path, one may pick such tuning parameter λ, that the minimizer has the same pattern as β. (Solution path for LASSO): Consider the LASSO optimization problem for ε = 0 and β = (10, 0, 0) ′ .

Figure 5 . 6 : 1 ( 3 ( 1 λ

 56131 Figure 5.6: Coordinates and the sign of βLASSO as the functions of λ > 0: βLASSO 1 (λ) (black curve), βLASSO2

4 . 4 . 5 . 5 . 4 . 4 .

 445544 (b) Thresholded penalized estimators can recover the pattern of β under much weaker condition than the noiseless recovery condition, see Section 5.Theorem Let Y = Xβ + ε where X ∈ R n×p is a fixed matrix, β ∈ R p and ε follows a symmetric distribution. Let pen be a polyhedral gauge. If β does not satisfy the noiseless recovery condition with respect to X and pen, then

Proposition 5 . 5 . 1 .

 551 Let pen be a real-valued polyhedral gauge, X ∈ R n×p and β ∈ R p . Then∃ y ∈ R n , ∃ β ∈ S X,λpen (y) such that β pen ∼ β ⇐⇒ ∃ y ∈ R n , ∃ β ∈ S X,λpen (y) such that ∂pen( β) ⊆ ∂pen(β).According to Propositions 4.4.2 and 5.5.1, if the pattern of β is not accessible with respect to X and λpen, i.e. there exists b ∈ R p such that Xb = Xβ and pen(b) < pen(β), then for any y ∈ R n , λ > 0, and β ∈ S X,λpen (y) we have ∂pen( β) ̸ ⊆ ∂pen(β). Consequently, no penalized nor thresholded penalized estimator can recover the pattern of β.

Theorem 5 . 5 . 1 .

 551 Let pen be a real-valued polyhedral gauge, X ∈ R n×p , β ∈ R p , and λ > 0. Assume that the uniform uniqueness holds, i.e. for any y ∈ R n , the set S X,λpen (y) consists of one element β(y). For ε ∈ R n and for r ∈ N + set y (r) = X(rβ) + ε. If pen(b) ≥ pen(β) for any b ∈ R p with Xb = Xβ, then there exists r 0 ∈ N + and τ ≥ 0 such that for all r ≥ r 0   ∂pen(b) ⊆ ∂pen(β) for any b ∈ B ∞ ( β (r) , τ ) ∃b 0 ∈ B ∞ ( β (r) , τ ) such that b 0 pen ∼ βConsequently, a τ -thresholded penalized estimator β τ (y (r) ) recovers the pattern of β.

Theorem 5 . 6 . 1 (

 561 Necessary and sufficient condition for uniform uniqueness). Let X ∈ R n×p and λ > 0. Let pen be a polyhedral gauge, i.e., pen(x) = max{0, u ′ 1 x, . . . , u ′ l x} for some u 1 , . . . , u l ∈ R p . For y ∈ R n consider the following optimization problem S X,λpen (y) = arg min b ∈ R p

Example 5 . 6 . 2 .

 562 [9, p. 19] Consider the following optimization problem arg min b

1 and 1 k

 1 ∈ row(X I ) , where I = [k] and 1 k = (1, . . . , 1) ′ ∈ R k . Moreover, the accessibility condition is satisfied with probability P X (min{∥γ∥ 1 : Xγ = X I 1 k } = k).

Figure 5 . 7 :

 57 Figure 5.7: Probability of the noiseless recovery condition and the accessibility condition as functions of the support size k. The highlited values of k are k 1 = 50/ log(150) ≈ 9.9 and k 2 = 100ρ DT (2/3) ≈ 47.8, which are the preimages of 0.5 for the noiseless recovery curve[START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (lasso)[END_REF] and the accessibility curve[START_REF] Donoho | Counting faces of randomly projected polytopes when the projection radically lowers dimension[END_REF][START_REF] Donoho | Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing[END_REF], respectively.

Figure 5 .

 5 Figure 5.7 provides these probabilities as functions of k.Figure 5.8 illustrates sign recovery properties by LASSO and thresholded LASSO for a particular

Figure 5 .

 5 Figure 5.7 provides these probabilities as functions of k.Figure 5.8 illustrates sign recovery properties by LASSO and thresholded LASSO for a particular observation of X ∈ R 100×150 , a particular observation of Y ∈ R 100 and for k-sparse β ∈ R 150 with β 1 = • • • = β k/2 = 20 and β k/2+1 = • • • = β k = -20. Our examples are given for k = 4 and k = 30. For the LASSO estimator, we consider the following setting:

Figure 5 . 8 :

 58 Figure 5.8: Illustrations of sign recovery by LASSO and thresholded LASSO. On the top, when k = 4, both the noiseless recovery condition and the accessibility condition hold. Thus, both LASSO and thresholded LASSO can recover the sign of β. With the large tuning parameter λ = 2 2 log(150) the sign of β is recovered both by LASSO and thresholded LASSO (top left). When the tuning parameter is small (computed by SURE), some null components of β are not correctly estimated at 0 (black points outside the x-axis), but there exists a threshold, for which the thresholded LASSO recovers the sign of β (top right). On the bottom, when k = 30, the accessibility condition holds but the noiseless recovery condition does not hold, thus thresholded LASSO can recover the sign of β but LASSO cannot. When the tuning parameter is large: λ = 2 2 log(150), both LASSO and thresholded LASSO fail to recover the sign of β (bottom left). When the tuning parameter is small, some null components of β are not correctly estimated at 0, but there exists a threshold, for which the thresholded LASSO recovers the sign of β (bottom right).

Figure 5 .

 5 Figure 5.9 provides both the probability of the accessibility condition and the probability of the noiseless recovery condition as functions of k.

Figure 5 . 9 :

 59 Figure 5.9: Probability of the noiseless recovery and the probability of the accessibility condition as functions of the number k of non-maximal components in absolute value. The value k = 50 [4] provides, approximately, the preimage of 0.5 for the accessibility curve.

Figure 5 . 10 : 1 1 . 5 . 8 . 1 .

 51011581 Figure 5.10: Illustrations of the pattern recovery by a penalized estimator and a thresholded penalized for the supremum norm. When k = 30, the accessibility condition holds, but the noiseless recovery condition does not hold. Thus, as illustrated on this picture, the recovery of the pattern of β requires thresholding the estimator.

Lemma 5 . 8 . 3 .

 583 Let X ∈ R n×p , Y ∈ R n , u 1 , . . . , u l ∈ R p and R ≥ 0. Then the optimization problem min ∥Y -Xb∥ 2 2 subject to the constraint u ′ 1 b ≤ R, . . . , u ′ l b ≤ R b ∈ P R . (5.8.1) has at least one solution. Proof. Let us set z = Xb in the optimization problem (5.8.1). Since the set XP R is a closed convex set, the problem of minimizing ∥Y -z∥ 2 2 subject to the constraint z ∈ XP R has a unique solution z ∈ XP R . Consequently, z = Xb for some b ∈ P R . Finally, b is a solution of the optimization problem (5.8.1).

Proof of 5 . 1 . 1 .

 511 Let m := inf b ∈ R p f (b). Note that 0 ≤ m ≤ f (0) = 1 2 ∥Y ∥2 2 . We want to show the existence of such β * ∈ R p that f (β * ) = m. If f (0) = m, then we are done. Assume then that f (0) = m + δ for some δ > 0. By the definition of infimum, one may consider such sequence

  which contradicts m = inf b ∈ R p f (b). Therefore the sequence (Xβ k ) k≥1 is convergent and thus (pen(β k )) k≥1 converges, too. Step 2: Existence of the minimizer: Denote g := lim k→∞ Xβ k and γ := lim k→∞ pen(β k ). Then we have m

.8. 3 )

 3 Now we construct β ∈ S X,λpen (y) with β ̸ = β. According to Lemma 5.8.1, ∂pen( β) = conv{u l : l ∈ I} where I = I pen ( β) = {l ∈ [k] : u ′ l β = pen( β)} and thus u ′

.8. 8 )

 8 Let α ∈ [0, ∞] be the limes superior of the sequence X β(y(r) )/rβ ) -λpen( β(y

  λpen(β) -λpen( β(y(r) )/r) r = 0.

Corollary 5 . 8 . 2 .Proposition 5 . 8 . 1 .

 582581 The noiseless pattern recovery is satisfied by β = 0 for every polyhedral gauge pen. Let 0 ̸ = β ∈ R p and I := {i ∈ [p] :

  .1.1) respectively, and the exponential density p = e(ϕ) = e ϕ-ψ(ϕ) = e ϕ /Z(ϕ). (6.1.2) Clearly, p > 0 and x∈X p(x)µ(x) = 1. For arbitrary real number c we have ψ(ϕ + c) = ψ(ϕ) + c, hence e(ϕ + c) = e(ϕ). (6.1.3)

. 1 .

 1 INTRODUCTION AND PRELIMINARIES 107 and the log-likelihood function isl e(ϕ) (x 1 , . . . , x n ) := log L e(ϕ) (x 1 , . . . , x n ) = n φ -ψ (ϕ) . (6.1.4)Of course, for every c ∈ R we have l e(ϕ+c) (x 1 , . . . , x n ) = l e(ϕ) (x 1 , . . . , x n ). (6.1.5)

Lemma 6 . 1 . 1 .Example 6 . 1 . 1 .

 611611 If MLE exists, then it is unique.Despite the boundedness (6.1.7), MLE may fail to exist, as shown by the following example.Let X = {0, 1}, µ ≡ 1, B = R X , n = 1 and x 1 = 1. Let a, b ∈ R and ϕ = a + b1 {1} . Then Z(ϕ) = e a (1 + e b ), e(ϕ) = e b1 {1} /(1 + e b ), and L e(ϕ) (x 1 ) = e(ϕ)(1) = e b /(1 + e b ). Thus, sup L e(ϕ) (x 1 ) = 1, but the supremum is not attained for any a, b ∈ R, so MLE does not exist in this case. On the other hand, if n = 3, x 1 = x 2 = 0, andx 3 = 1, then L e(ϕ) (x 1 , x 2 , x 3 ) = e b /(1 + e b )3 . By calculus, the maximum is attained when e b = 1/2, therefore p = (2 -1 {1} )/3 is the MLE in this case.

Corollary 6 . 2 . 1 .

 621 If {x 1 , . . . , x n } is of uniqueness for B then MLE exists for e(B) and x 1 , . . . , x n .

a

  j r j ∈ B k + .Let χ = -(sign(a 1 ), . . . , sign(a k )), where, say, sign(0) = 1. Obviously, χ ∈ Q k , and since r(χ) ≥ 0, we get

Example 6 . 3 . 4 .

 634 1 , . . . , x n } is a set of uniqueness for B k + . Let x ∈ Q k be arbitrary. By Theorem 6.3.3, MLE for e B k and {x, -x} exists.

Example 6 . 3 . 5 .

 635 If MLE fails to exist for e(B k ) and x 1 , . . . , x n ∈ Q k , then the following analysis may shed some light on Theorem 6.2.5. Let J = {j ∈ {1, . . . , k} : {r j (x 1 ), . . . , r j (x n )} = {-1, 1}}, J ′ = {1, . . . , k} \ J.

→ e - 2 1 5 ) 6 . 3 . 6 .Lemma 6 . 3 . 2 . 1 kk log 2 + 1 ≤

 25636632121 -b , as k → ∞. (6.3.Corollary log 2 k is a sharp threshold of the sample size for the existence of MLE for e(B k ) and i.i.d. uniform samples on Q k . Proof. Let ε ∈ (0, 1) and (the sample size) n = n(k) < (1 -ε) log 2 k. Then,P (ν uniq < n) ≤ P (ν uniq < (1 -ε) log 2 k) .For every b ∈ R by the equation in (6.3.5) we have lim supk→∞ P (ν uniq < (1 -ε) log 2 k) ≤ lim sup k→∞ P (ν uniq < log 2 k + b) = e -2 1-b .Since b is arbitrary, we conclude that lim sup k→∞ P (ν uniq < n(k)) = 0. Analogously, for thesample size n = n(k) > (1 + ε) log 2 k we get lim inf k→∞ P (ν uniq > n(k)) = 1,which ends the proof.The above is in stark contrast to Corollary 6.3.2, as summarized in Remark 6.3.2. Indeed, in the present setting we have K = |Q k | = 2 k , so the sharp threshold for the sample size needed for the existence of MLE is log 2 log 2 K. The following result on the expectation of ν uniq agrees well with the sharp threshold. Let ν uniq be as in Corollary 6.3.5. Let H k = k i=1 be the k-th harmonic number. Then, H E(ν uniq ) < H k log 2 + 2, k = 1, 2, . . . .

Theorem 6 . 5 . 1 .

 651 MLE exists for e(Bk k-1 ) and x 1 , . . . , x n ∈ Q k if and only if E or O ⊂ {x 1 , . . . , x n }.Proof. Thanks to Theorem 6.2.2, we only need to characterize the sets of uniqueness for B k k-1 +

  r j (x) ̸ = r j (y)}| = 1} .Let U = {x 1 , . . . , x n }. Assume that U is a set of uniqueness. Let e ∈ E and o ∈ O. The hypercube graph G Q k is connected, so there exists a path (e, v 1 , v 2 , . . . , v 2p , o) in G Q k . Then 1 {e,v 1 } + 1 {v 2 ,v 3 } + . . . + 1 {v 2p ,o} -1 {v 1 ,v 2 } + 1 {v 3 ,v 4 } + . . . + 1 {v 2p-1 ,v 2p } (6.5.1) = 1 {e} + 1 {o} (6.5.2)is a nontrivial nonnegative function on Q k . Therefore, we must have {e, o} ∩ U ̸ = ∅. Then we easily conclude that E ⊂ U or O ⊂ U .

6. 6 . 3 Proof of Lemma 6 . 4 . 1 By ( 6 . 4 . 1 )

 63641641 , each G ∈ G N appears in G N,c with probability p c (G) = e ϕc(G)-ψ(ϕc) . Then,p r,s = P ((r, s) ∈ E (G)) = G∈G N (r,s)∈E(G) e ϕc(G) G∈G N e ϕc(G) = G∈G N (r,s)∈E(G) e ϕc(G) G∈G N (r,s)∈E(G) e ϕc(G) + G∈G N (r,s) / ∈E(G) e ϕc(G) G N 10 = {G ∈ G N : S 1 ⊂ G, S 2 ̸ ⊂ G} , G N 02 = {G ∈ G N : S 1 ̸ ⊂ G, S 2 ⊂ G} , G N 00 = {G ∈ G N : S 1 ̸ ⊂ G, S 2 ̸ ⊂ G} .

φ

  < max X ϕ. Recall that ϕ ∈ B if and only if there is θ ∈ R d such that ϕ = θ • T . Then min x∈X ϕ(x) = min y∈S θ • y = min y∈C θ • y, max x∈X ϕ(x) = max y∈C θ • y, and, of course, φ = θ • T . Therefore the existence of MLE for x 1 , ..., x n and e(B) is equivalent to T being in the relative interior of C.

  c,c corresponding to c ∈ C. In other words, L * C (G) := L(G) + E C

Theorem 7 . 2 . 1 .

 721 Let (X 1 , . . . , X n ) be a Gaussian graphical model with an underlying graph G being a tree rooted in C = {c}. Assume that its covariance matrix Σ = (σ i,j ) satisfiesσ i,j = |AN(i) ∩ AN(j)|.Then the precision matrixK = Σ -1 of (X 1 , . . . , X n ) is equal to L * c = L * c (G).

Proof.

  It suffices to prove that L * c (G) • Σ = I n . At first consider the case i = j = c. Observe that for any 1 ≤ k ≤ n we have σ c,k = 1. Therefore:(L * c Σ) c,c = n k=1 l * c,k σ k,c = l * c,c σ c,j + k∼c l * c,k σ k,c = (deg(c) + 1) + deg(c) • (-1) • 1 = 1. Now consider i = j ̸ = c: (L * c Σ) i,i = n k=1 l * i,k σ k,i = l * i,i σ i,i + k∼i k∈AN(i) l * i,k σ k,i + k∼i i∈AN(k) l * i,k σ k,i = = deg(i)σ i,i -(σ i,i -1) -(deg(i) -1)σ i,i = 1.Now we will prove that the outside of the diagonal of (L * Σ) consists only of zeros. Observe that for i = c, j ̸ = c we have:(L * c Σ) i,j = n k=1 l * i,k σ k,j = l * i,i σ i,j + k∼i k / ∈AN(j) l * i,k σ k,j + k∼i k∈AN(j) l * i,k σ k,j = = (deg(i) + 1) • 1 -(deg(i) -1) • 1 -1 • 2 = 0.On the other hand, if j = c, i ̸ = c, then: of the graph Laplacian sums up to 0. Now we let i, j ̸ = c, i ̸ = j. If i ∈ AN(j), then:(L * c Σ) i,j = n k=1 l * i,k σ k,j = n k=1 l i,k • σ k,j == deg(i) • σ i,j -(deg(i) -2)σ i,j -1 • (σ i,j -1) -1 • (σ i,j + 1) = 0.

- 1 Theorem 7 . 3 . 1 .= 3 • 1 + (- 1 ) 1 Σ

 17313111 The description of the inverse of L * (G) for general G is much harder for G not being a tree. So far we are not able to present a general formula, thus we consider examples of such inverse for specific classes of graphs G. Below we show some examples of n × Σ = [L * 1 (G)] -1 for cycles C n and complete graphs K n with the root C = {1}: The inverse matrix Σ = (σ i,j ) of L * (C n ) is a symmetric matrix satisfyingσ i,j = 1 + (i -1)(n -j + 1) n , for i ≤ j.Proof. Again, we show that L * 1 Σ = I n , we assume that the cycle is 1→ 2 → 3 → . . . → n → 1 and take C = {1}. Then (L * 1 Σ) c,c = n k=1 l c,k σ k,c = n k=1 l c,k = 1.If j ̸ = c, then we have(L * 1 Σ) c,j = n k=1 l c,k σ k,j = l c,c σ c,j + l c,2 σ 2,j + l c,n σ n,j = ) i,c = n k=1 l i,k σ k,c = n k=1 l i,k = 0.

3

 3 The augmented Laplacian (and its inverse) depends on the choice of a root set C ⊂ V . Below we consider four choices of a root in D a,b and their augmented Laplacians:L * in := L(D a,b ) + E c for C = {c}, c ∈ V A , L * * in := L(D a,b ) + c∈A E c for C = V A , L * ex := L(D a,b ) + E c for C = {c}, c ∈ V B , L * * ex := L(D a,b ) + c∈A E c for C = V B .

  c) Gaussian random variables with covariance Σ To observe the analogy of the previous examples for the cycle graph C n , we may note that replacing the edge (n, 1) with (n, n + 1) (with n + 1 / ∈ V ) gives a path graph [c = 1] -[2] -. . . -[n + 1]).

1 - 1 nn- 1

 111 and α = β + 1 n . Titre : Aspects Géométriques et Combinatoires de Modeles Statistiques. Mot clés : régression linéaire, SLOPE, polyédre, MLE, graphe aléatoire Résumé : Cette thèse traite des applications de la combinatoire et de la géométrie discrète aux statistiques modernes. Ça commence par les méthodes de régression linéaire pénalisée, en particulier par l'estimateur SLOPE. Nous examinons la notion de schéma SLOPE, qui conserve l'information sur le support, le signe et le classement des coefficients de régression. Nous fournissons les conditions pour la récupèration de l'ensemble des variables pertinentes et des groupes en cas orthogonal. Pour une classe générale de matrices de planification fixes, nous fournissons la condition d'irreprésentabilité du SLOPE, qui est nécessaire et suffisante pour la recouvrement du schéma dans le cadre non-bruité et nous illustrons cette propriété géométrique-ment. Après nous considérons le cas asymptotique. Egalement, nous étudions la classe plus large des estimateurs pénalisés, appelés les jauges polyédriques. La seconde application s'articule autour de l'existence de l'estimateur MLE pour les familles exponentielles discrètes. Par la suite, nous inspectons la taille des échantillons indépendants identiquement distribués qui est nécessaire pour assurer son existence avec une grande probabilité. Pour cela, nous utilisons les notions issues de l'analyse des hypercubes discrets et appliquons nos résultats à des graphes aléatoires.

  celu jest pojęcie zbiorów jednoznaczności, czyli takich pozdbiorów przestrzeni stanów, na którym funkcja z danej klasy jest równa zero wtedy i tylko wtedy, gdy jest ona stale równa zero na całej przestrzeni. Następnie wykorzystujemy nasze kryterium do klasy funkcji Rademachera i Walsha oraz do modelu wykładniczych grafów losowych (ERGM). W rozdziale siódmym rozważamy powiązania między gaussowskimi modelami graficznymi, a macierzami kowariancji zdyskretyzowanych procesów Wienera. W tym celu używamy laplasjanu grafu, który opisuje strukturę niezależności warunkowej zadanego modelu.Récemment nous pouvons observer l'émergence rapide de données massives, tant au niveau du nombre d'observations que du nombre de variables explicatives à mesurer. En raison de l'augmentation de l'influence des big data sur la vie quotidienne, la statistique mathématique

	Résumé
	1. Pattern recovery and signal denoising by SLOPE when the design matrix is orthogo-
	nal [165] (z P. Graczykiem, B. Kołodziejkiem i M. Wilczyńskim), Probability and Mathe-
	matical Statistics, 42(2):283-302, 2022
	2. Pattern recovery by SLOPE [23] (z M. Bogdan, X. Dupuisem, P. Graczykiem, B.

Kołodziejkiem, P. Tardivelem i M. Wilczyńskim), wysłany do recenzji https://arxiv. org/pdf/2203.12086.pdf 3. Pattern Recovery in Penalized and Thresholded Estimation and its Geometry

[START_REF] Graczyk | Pattern recovery in penalized and thresholded estimation and its geometry[END_REF]

(z P. Graczykiem, U. Schneider i P. Tardivelem), w przygotowaniu, https://hal.science/ hal-03262087v2/document 4. Maximum likelihood estimation for discrete exponential families and random graphs [22] (z K. Bogdanem i M. Bosym), ALEA, 19, 1045-1070 (2022) 5. Remarks on Laplacian of Graphical Models in Various Graphs [164], Proceedings, GSI 2021, Paris, France, July 21-23, 2021 Plan niniejszej rozprawy jest następujący: Rozdział drugi zawiera podstawowe pojęcia dotyczące wyników używanych w dalszych rozdziałach. Rozdział trzeci skupia się na metodzie SLOPE w przypadku ortogonalnej macierzy eksperymentu, tzn. X ′ X = nI p . W tym przypadku wprowadzamy również nowe wyniki dotyczące mocnej zgodności estymatora SLOPE i jego wzorca. Następnie ilustrujemy numerycznie zastosowanie SLOPE w klasteryzacji do odtwarzania sygnałów o wysokiej frekwencji. W rozdziale czwartym omawiamy odtwarzanie wzorca SLOPE bez ograniczeń z poprzedniego rozdziału. Pokazujemy, że odtwarzanie wzorca można scharakteryzować za pomocą dwóch warunków, które nazywamy 'positivity condition' oraz 'subdifferential condition'. Następnie wprowadzamy 'irrepresentability condition' dla SLOPE, który jest uogólnieniem 'irrepresentability condition' dla LASSO [82, 41] wraz z jego geometryczną interpretacją. W dalszej kolejności podajemy bardziej wyrafinowane warunki asymptotyczne na zgodność oraz mocną zgodność estymatora SLOPE i jego wzorca. Rozdział piąty dotyczy odtwarzania wzorca w ogólniejszym przypadku kar postaci polyhedral gauges. Wyprowadzamy warunki konieczne i dostateczne na jednorodną jednoznaczność estymatora, tzn. jego jednoznaczność dla dowolnej wartości Y ∈ R n . Również pokazujemy kryteria na odtwarzanie wzorca dla podanych estymatorów i ich progowanych (thresholded) odmian. Ponadto, uogólniamy irrepresentability condition na penalizację poprzez polyhedral gauges oraz opisujemy relacje pomiędzy wzorcem, a modelem rozważanym w publikacji Vaitera i in. [182]. W rozdziale szóstym podajemy konieczne i dostateczne warunki na istnienie estymatora największej wiarogodności (MLE) w dyskretnych rodzinach wykładniczych. Głównym narzędziem PL.5. PLAN ROZPRAWY xxi używanym w tym et la science des données gagnent un intérêt croissant dans le domaine des mathématiques et des sciences appliquées. Cependant, elles ne doivent pas être considérées séparément des autres branches des mathématiques. Dans cette thèse, nous nous attachons à proposer de nouveaux liens entre la statistique moderne, la géométrie et la combinatoire. Nous commençons notre discussion par les espaces euclidiens de dimension finie. Nous prêtons attention aux situations où l'espace des valeurs possibles de l'estimateur peut être partitionné en un nombre fini de sousensembles tels qu'il existe une correspondance bijective entre les sous-ensembles et les propriétés spécifiques des estimations. Cette prédiction des propriétés d'une observation est mieux connue sous le nom de problème de classification

  OLS a gagné sa popularité dans le cas n ≥ p en raison de sa simplicité, de son efficacité pour un un nombre relativement bas de variables[33, str. 4] et de nombreuses propriétés statistiques utiles. Par exemple, si la matrice X

	xxiv	CHAPTER 0. INTRODUCTION (INTRODUCTION IN FRENCH)
		xxiii

Dans le modèle de régression linéaire multiple ayant n observations et p variables explicatives, nous supposons que le vecteur de réponse Y = (y 1 , . . . , y n ) ′ ∈ R n est de la forme Y = Xβ + ε, où X ∈ R n×p est une matrice de planification, β = (β 1 , . . . , β p ) ′ ∈ R p est un vecteur inconnu de coefficients de régression et ε = (ε 1 , . . . , ε n ) ′ ∈ R n est un bruit aléatoire. L'objectif principal de la régression est d'estimer β et de retrouver ses propriétés essentielles. La abondante classe de problèmes de choix du sous-ensemble de coordonnées de β, qui est le plus approprié pour nos demandes sur l'estimation et sa parcimonie, est mieux connue sous le nom de problème de sélection de modèle. Pour une étude sur le problème du sélection de modèle, cf.

[START_REF] Claeskens | Statistical model choice[END_REF]

. Dans la plupart de nos discussions, nous supposons que l'erreur ε a une distribution symétrique et continue. Certains de nos résultats couvrent le cas sans erreur de ε ≡ 0, qui est outil pour établir des résultats asymptotiques. En raison de l'abondance des connaissances existantes sur la distribution gaussienne, nous sommes également en mesure de présenter des résultats plus efficaces sur le comportement des estimateurs avec gausienne ε ∼ N (0, σ 2 I n ). La méthode de régression linéaire la plus classique, la méthode des moindres carrés ordinaires (OLS), a été proposée par Legendre au début du XIXe siècle

[START_REF] Barbin | Histoires de probabilités et de statistiques[END_REF][START_REF] Hald | A history of parametric statistical inference from Bernoulli to Fisher, 1713-1935[END_REF][START_REF] Legendre | Nouvelles méthodes pour la détermination des orbites des comètes. Nineteenth Century Collections Online[END_REF]

. L'estimateur OLS est défini comme le minimiseur de la somme des carrés des résidus, à savoir βOLS := arg min b ∈R p ∥Y -Xb∥ 2 . ′ X est inversible et Cov(Y ) = σ 2 I n pour σ > 0, alors les OLS constituent le meilleur estimateur linéaire sans biais (BLUE) [1, Theorem 2.7.1.] de β. Dans ce cas, la formule exacte de βOLS peut être calculée à partir de l'équation suivante pour βOLS peut être facilement déduite

[1, str. 28]

: βOLS

  4 et 4.6 se concentrent sur la forte consistance de SLOPE et de son schéma. Il est important de rappeler que si le vecteur d'accord Λ est constant (LASSO comme cas particulier de SLOPE), alors le SLOPE schéma n'est pas consistant, même si l'hypothèse sur la consistance du vecteur de signe est satisfaite. A titre d'exemple, les clusters d'un vrai vecteur de paramètres, qui sont contenus dans un SLOPE schéma, ne sont pas préservés par LASSO.

	FR.2. SLOPE			xxvii
		β 2
		OLS
	(-2,1)	SLOPE
		SLOPE	OLS	β 1
		SLOPE
		OLS

Comme autre remarque intéressante, on peut mentionner que la forte consistance de LASSO (ou SLOPE) n'implique pas nécessairement la forte consistance de son signe (de son schéma SLOPE). Comme contre-exemple, on peut remarquer que la séquence ((1/n, 1/n, -1/n, -1/n) ′ ) n≥1 converge vers (0, 0, 0, 0) ′ , alors que son signe et son schéma SLOPE sont égaux (1, 1, -1, -1) ′ pour tout n positif.

  Cela nous permet de dériver la fonction de vraisemblance et de log-vraisemblance. La fonction de log-vraisemblance est strictement concave. Par conséquent, si l'estimateur du maximum

de vraisemblance (MLE) existe, il est unique. Malgré le caractère borné de la fonction de vraisemblance, l'MLE peut ne pas exister. Notre objectif principal est d'établir une nouvelle caractérisation de l'existence de l'MLE et de l'appliquer à des familles spécifiques, les familles exponentielles couvertes par les fonctions de Rademacher et de Walsh, et les familles exponentielles de graphes aléatoires. L'outil clé dans notre discussion est la notion nouvellement introduite d'ensemble d'unicité, c'est-à-dire un tel ensemble U ⊂ X, que ϕ = 0 est la seule fonction d'une classe donnée de fonctions qui est égale à zéro sur U . Pour être plus précis, nous montrons que le MLE pour e(B) et l'échantillon i.i.d. x 1 , x 2 , . . . , x n ∈ X existe si et seulement si {x 1 , . . . , x n } est l'ensemble d'unicité pour le cône non négatif B + := {ϕ ∈ B : ϕ ≥ 0}. Nous reformulons également ce critère sous la forme d'un problème de programmation linéaire.

  β

	SLOPE p	(n)

′

be the SLOPE estimator defined by (3.4.2). With the above notation we present the main result of this section. Theorem 3.4.3. Assume that

Table 3 .

 3 3.2 are verified and the proof is complete. 1: Comparison of MSE between different regression methods.

	OLS	LASSO-CV LASSO-LS SLOPE-LS
	M SE(β, •) 613.6797	426.3705	171.7957	20.74967

  where (ε i ) i are i.i.d. centered with finite variance.

	B1. A sequence of design matrices X 1 , X 2 , . . . satisfies the condition
	1 n	X ′ n X n

P

-→ C,

(4.5.4)

where C is a deterministic positive definite symmetric p × p matrix.

B2. For each j = 1, . . . , p, max i=1,...,n |X

  Then, applying Theorem 2.3.18 on the probability space (Ω, F, P X ) to our sequence, we obtain that for j = 1, . . . , n,

1), (4.6.3), (4.6.4) and (4.6.5) coincide with (2.3.4), (2.3.6), (2.3.7) and (2.3.8), respectively. Let P( •|(X n ) n ) be a regular conditional probability.

  Theorem 4.6.2. Assume that Yn = X n β + ε n , where β ∈ R p , ε n = (ε 1 , . . . , ε n ) ′ with ε i i.i.d.,centered and having finite variance. Suppose (4.6.8) and that (ε n ) n and (X n ) n are independent. Let Λ n = (λ Proof of Theorem 4.6.2. The assumption (4.6.8) implies that the matrix X ′ n X n is positive definite for large n almost surely and hence ensures that ker(X n ) = {0}. It implies the uniqueness of the SLOPE solution.

	(n) 1 , . . . , λ	(n)
	If β ̸ = 0, then
				βSLOPE n	a.s. -→ β ⇐⇒ lim n→∞	(n) 1 n λ	= 0 .	(4.6.9)
	If β = 0 and lim n→∞	λ	(n) 1 n = 0, then βSLOPE n	a.s. -→ 0.
	By Proposition 4.2.1, βSLOPE n	is the SLOPE estimator of β in a linear regression model

p ) ′ . Then, for large n, S Xn,Λn (Y n ) = { βSLOPE n } almost surely.

4.9.1 Proof of Proposition 4.2.1

  Note that if M = 0, then the statement is valid by (4.2.1). Thus, we may later assume that M ̸ = 0. To simplify the notation, we write Λ instead of Λ M . The elements of Λ are denoted by Λ l , l = 1, . . . , k. Let k = ∥M ∥ ∞ . Before proving Proposition 4.2.1, note that, by assumption, there exists s ∈ R k+ such that b = U M s. Consequently, |b| ↓ = U |M| ↓ s and thus

  then the first part of the proof establishes that ψ( M ) = ψ(M ) and thus M = M .

	Recall that J * Λ (x) ≤ 1 if and only if
	|x| (1) + . . . |x| (j) ≤ λ 1 + . . . + λ j ,	j = 1, . . . , p.	(4.9.2)
	The following result follows from the proof of Proposition 4.2.1.

Proposition 4.9.1. Assume x ∈ R p satisfies J * Λ (x) ≤ 1 and let b ∈ R p . Then x belongs to ∂J Λ (b) if and only if the following three conditions hold true:

jump(x) = jump(z).

  are equal if and only if For instance, if x = (1.45, 1.45, 0.56, 0.56, -0.45, 0.35) ′ then jump(x) = (→, ↘, →, ↘, ↗) ′ . Figure5.4 compares the projections onto B * with the jump sets of β for p = 2 and orthogonal X.

  In the setting of Example 6.2.1 we have {

	and so sup p∈e(B) sup p∈e(B) Motivated by Lemma 6.2.1, we define Therefore, {x 1 , . . . , x n } B + = ϕ -1 ({0}), where the intersection is taken over all ϕ ∈ B + such that ϕ(x 1 ) = . . . = ϕ(x n ) = 0. Thus for (6.2.3) all ϕ ∈ B Example 6.2.4.

l p (x 1 , . . . , x n ) ≤ sup p∈e( B) l p (x 1 , . . . , x n ).

Let δ ∈ B + and k be as in (6.2.1). Using (6.2.1) and (6.2.2),

l e(ϕ-kδ) (x 1 , . . . , x n ) → l e( ϕ) (x 1 , . . . , x n ), as k → ∞. l p (x 1 , . . . , x n ) ≥ sup p∈e( B)

l p (x 1 , . . . , x n ). + , if ϕ vanishes on {x 1 , . . . , x n }, then it vanishes on {x 1 , . . . , x n } B + , and the latter is the largest such set. Put differently, if there is δ ∈ B + such that δ(x 1 ) = . . . = δ(x n ) = 0 but δ(x) > 0, then x / ∈ {x 1 , . . . , x n } B + , and conversely. In particular, U ⊂ X is set of uniqueness for B + if and only if U B + = X .

  Corollary 6.3.3. MLE for e(B k ) and x 1 , . . . , x n ∈ Q k exists if and only if {x 1 , . . . , x n } has a nonempty intersection with each half-cube. The proof of Corollary 6.3.3 is immediate from Theorem 6.3.3 and the discussion above.

  2 ,s 2 = p r 1 ,s 1 p r 2 ,s 2 .

	
	 	i∈Iq	1 H + i

6.6.5 Proof of Lemma 6.5.1 Proof. Consider the positive half-cubes H + 1 , . . . , H + k . Let B = Lin

and {R ij } i,j are independent. By Theorem 6.3.3,

Acknowledgements Acknowledgments: We are grateful to Małgorzata Bogdan, Piotr Ciołek, Persi Diaconis, Hélène Massam, Sumit Mukherjee, Krzysztof Oleszkiewicz, Krzysztof Samotij, Maciej Wilczyński and anonymous referees for comments, corrections, references and discussion.

gratitude to my supervisors Professors Maciej Wilczyński and Piotr Graczyk for their guidance and support during the time of my PhD studies. Thank you for the opportunity of sharing the studies between Wrocław and Angers and for your encouragement to submit the results of our long and fruitful discussions for publication. I would like to thank Dr. Patrick Tardivel for agreeing to be the co-encadrant in France and to extend our research to discussions in Dijon. Many thanks to Professor Krzysztof Bogdan for his guidance from the beginning of the PhD studies, for many inspiring discussions and for initiating the co-tutelle agreement with Université d'Angers. , especially to my Parents, who taught me how to count and think and supported my through the PhD, even during the harder times. Thanks are also due to all the people whom I've met while pursuing my hobbies, especially to the Lingustics Olympiad community and the Vytautas' Light Cavalry. The major part of the research was supported by a French Government Scholarship. The research in Wrocław University of Science and Technology was supported in part by grant 049M/0010/19 from WUST. The research in Université d'Angers was supported by Centre Henri Lebesgue, program ANR-11-LABX-0020-0. Thank you for helping me to focus on research.

(ii) Assume A, B1, B2 and C. The sequence n -1/2 π [START_REF] Akaike | A new look at the statistical model identification[END_REF] n n converges in distribution to a Gaussian vector V with (i') If M ̸ = 0, then 1 α n π (1) n a.s.

-→ CU M (U ′ M CU M ) -1 Λ.

(ii') If lim n→∞ α n / √ n log log n = ∞, then α -1 n π

(2) n a.s.

-→ 0.

Proof. (i) Assumption B implies that

(ii) When β = U M s 0 , then the linear regression model takes the form Y n = X n s 0 + ε n . Since P n is the projection matrix onto col( X n ), we have (I n -P n ) X n = 0. Thus,

By assumption B we have,

Thus, by Lemma 4.9.3 and Slutsky's theorem, we obtain (ii).

(iii) follows directly from (ii). Assumption B' implies that n -1 X ′ n X n a.s.

-→ C and thus (i') is proven in the same way as (i). (ii') follows from (4.9.9).

Proof of Theorem 4.5.3. (a) is a direct consequence of Lemmas 4.9.4 and 4.9.5. Since positivity condition is satisfied for large n with high probability, for (b) we have with M = patt(β),

n π n ∈ ∂J Λ (M ) (4.9.12)

where in the last inequality we use Portmanteau Theorem, assumption (4.5.5) and the fact that the sequence (α -1 n π n ) n converges in distribution to CU M (U ′ M CU M ) -1 if and only if α n / √ n → ∞. Condition (4.5.6) implies that CU M (U ′ M CU M ) -1 ∈ ∂J Λ (M ). Since (α -1 n π n ) n converges in probability to CU M (U ′ M CU M ) -1 , the necessity of this condition is explained by (4.9.12).

Examples of polyhedral gauges and their patterns

Pattern for the ℓ 1 norm (LASSO):

Subdifferentials ∂∥.∥ 1 (x) = ∂∥.∥ 1 (z) are equal if and only if sign(x) = sign(z). For instance, if x = (1.45, -0.38, 1.56, 0, -2.76) ′ , then sign(x) = (1, -1, 1, 0, -1) ′ . Figure 5.1 illustrates the comparison between projections onto B * and the signs of β LASSO for p = 2 and orthogonal X.

(0, 1) (0, 1) (0, 1) (

(-1, -1)

(0, 0) The picture on the right presents sign equivalence classes of vectors in R 2 .

Pattern for the ℓ ∞ norm:

The vector sign ∞ (x) ∈ {-1, * , 1} p is defined as follows

The notation * represents components that are not maximal in absolute value. Subdifferentials ∂∥.∥ ∞ (x) = ∂∥.∥ ∞ (z) are equal if and only if sign ∞ (x) = sign ∞ (z). For instance, if x = (1.45, 1.45, 0.56, 0, -1.45) ′ then sign ∞ (x) = (1, 1, * , * , -1) ′ . Figure 5.2 illustrates the comparison between projections onto B * and the pattern equivalence classes of β for pen(x) = ∥x∥ ∞ , p = 2 and orthogonal X.

Pattern for the SLOPE norm:

Conversely, let F be a non-empty face of P . Then F = ∂ϕ(x) for some x ∈ R p .

Proof. The fact that ∂ϕ(x) = conv{u l : l ∈ I ϕ (x)} can be found in [101, p. 259]. To prove the second equality, we consider the following. If l ∈ I ϕ (x), then by definition of I ϕ (x), v ′ l x = ϕ(x) and thus v l ∈ {s ∈ P : s ′ x = ϕ(x)}. Since the latter set is convex, one may deduce that

For the other inclusion, assume that s ∈ P is such that s / ∈ conv{v l : l ∈ I ϕ (x)}. Then we have

, we also get

Consequently, s ′ x ̸ = ϕ(x) and thus

Therefore, ∂ϕ(x) = conv{v l : l ∈ I ϕ (x)} = {s ∈ P : s ′ x = ϕ(x)}. Now we show that the subdifferentials of ϕ are the non-empty faces of P . Let x ∈ R p . By definition of ϕ, v ′ l x ≤ ϕ(x) for every l ∈ [k], so the inequality x ′ s ≤ ϕ(x) holds for all s ∈ P . This implies that ∂ϕ(x) is a non-empty face of P . Now we show that any non-empty face F of P is equal to ∂ϕ(a) for some a ∈ R p . Let F = {s ∈ P : a ′ s = c} be a non-empty face of P where a ∈ R p , c ∈ R and a ′ s ≤ c is a valid inequality for all s ∈ P . We prove that F = ∂ϕ(a). Indeed, take any s ∈ F . We get a ′ s = c as well as a ′ s ≤ ϕ(a) as shown above, implying that c ≤ ϕ(a). Analogously, for any s ∈ ∂ϕ(a), a ′ s = ϕ(a) as well as a ′ s ≤ c since ∂ϕ(a) ⊆ P , yielding ϕ(a) ≤ c. Therefore one may deduce that ϕ(a) = c and thus F = ∂ϕ(a). The following lemma, also needed to show Theorem 5.6.1, states that the fitted values are unique over all non-unique solutions of the penalized problem for a given y. It is a generalization of Lemma 1 in [START_REF] Tibshirani | The lasso problem and uniqueness[END_REF], which shows this fact for the special case of the LASSO.

Lemma 5.8.2. Let X ∈ R n×p , y ∈ R n , λ > 0 and pen be a polyhedral gauge. Then X β = X β and pen( β) = pen( β) for all β, β ∈ S X,pen (Y ).

Proof. Assume that X β ̸ = X β for some β, β ∈ S X,λpen (Y ) and let β = (

Moreover, by convexity of pen, we have pen( β)

which contradicts both β and β being minimizers. Finally, X β = X β implies that pen( β) = pen( β).

which proves what was claimed.

2) Assume that 0 / ∈ aff{u l :

l ∈ I}, we also have ker(X) ⊆ v ⊥ and (col(U )) ⊥ ⊆ v ⊥ , yielding a contradiction and proving the claim.

( ⇐= ) Assume that there exists y ∈ R n such that β, β ∈ S X,λpen (y) with β ̸ = β. Then

. Consequently, one may deduce that row(X) intersects the face ∂pen( β) ∩ ∂pen( β). Let F * = conv{u l : l ∈ I * } be a face of ∂pen( β) ∩ ∂pen( β) of the smallest dimension among all faces of ∂pen( β) ∩ ∂pen( β) intersecting row(X). By minimality of dim(F * ), row(X) intersects the relative interior of F * , namely, there exists z ∈ R n such that v = X ′ z lies in F * , but not on a proper face of F * . Now we will show that if dim(F * ) ≥ dim(ker(X)), then row(X) intersects a proper face of F * , yielding a contradiction. For this, we start with observing that dim(F * ) = dim(aff{u l : l ∈ I * }) and that we can write the affine space aff{u l :

Therefore, h ∈ ker(X) ∩ col(U * ) ⊥ , where

Assume that dim(F * ) ≥ dim(ker(X)). Then dim(row(X)) + dim(col( U * )) ≥ rk(X) + dim(ker(X)) = p.

If row(X) ∩ col( U * ) = {0}, then R p = row(X) ⊕ col( U * ). However, the last equation cannot hold since row(X) = ker(X) ⊥ ⊆ h ⊥ and col( U * ) ⊆ col(U * ) ⊆ h ⊥ , where h ̸ = 0. Consequently,

intersects the relative interior of

But then also row(X) intersects a proper face of F * , which yields the required contradiction.

Proof of Proposition 4.4.2

The following lemma can be seen as a generalization of Proposition 4.1 from [START_REF] Gilbert | On the solution uniqueness characterization in the l1 norm and polyhedral gauge recovery[END_REF] from the ℓ 1 norm to all convex functions. By definition of the subdifferential it may be shown that ∂ι β (β) = row(X). Therefore we can deduce that the implication (5.8.4) holds for any b ∈ R p if and only if

Proof of Proposition 4.4.2. By Lemma 5.8.4, the geometric characterization of accessible patterns is equivalent to the analytic one. We show the geometric characterization.

( =⇒ ) If the pattern of β is accessible with respect to X and λpen, then there exists y ∈ R n and β ∈ S X,λpen (y) such that β pen ∼ β. Because β ∈ S X,λpen (Y ), we get 1 λ X ′ (y -X β) ∈ ∂pen( β) = ∂pen(β), thus row(X) intersects ∂pen(β).

( ⇐= ) If row(X) intersects the face ∂pen(β), then there exists z ∈ R n such that X ′ z ∈ ∂pen(β). For y = Xβ + λz, we have 1 λ X ′ (y -Xβ) = X ′ z ∈ ∂pen(β), so that β ∈ S X,λpen (y), and the pattern of β is accessible with respect to X and λpen. Proof. Let s ∈ ∂ϕ(x) = ∂ϕ(v). Since s is both a subgradient at x and at v, the following inequalities hold

Multiplying the first inequality by α, the second by (1 -α) and adding them, we get

Using the convexity of ϕ, we arrive at

By Lemma 5.8.1 we have ∂ϕ(x) = conv{u l : l ∈ I}, where

) and the claim follows.

Lemma 5.8.6. Let X ∈ R n×p and β ∈ R p . Then the following set is convex

Note that V β may be an empty set.

Proof. Assume that V β ̸ = ∅. Let y, y ∈ V β . Then there exist λ > 0 and λ > 0 such that β ∈ S X,λpen (y) and β ∈ S X, λpen ( y) with ∂pen( β) = ∂pen( β) = ∂pen(β). Consequently,

Let α ∈ (0, 1) and y = αy + (1 -α) y. Define λ = αλ + (1 -α) λ and β = α β + (1 -α) β. Now we show that y ∈ V β . Indeed, observe that

By Lemma 5.8.5, ∂pen( β) = ∂pen(α β + (1 -α) β) = ∂pen(β), then also β ∈ S X, λpen (y), which proves the claim.

Proof of Theorem 5.4.4. Assume that the noiseless recovery condition does not hold for β. Then Xβ / ∈ V β , where V β is defined as in Lemma 5.8.6. Consequently, by convexity of V β , for any

Consequently, 1 2

Proof of Proposition 5.5.1

Proof of Proposition 5.5.1. We only need to prove the implication ( ⇐= ), as the other implication is obvious. Assume that ∂pen( β) ⊆ ∂pen(β). Since β ∈ S X,λpen (y), we have

Consequently, row(X) intersects ∂pen(β) which implies that the pattern of β is accessible with respect to X and pen by Proposition 4.4.2. Consequently, there exists y ∈ R n and there exists β ∈ S X,λpen (y) for which β pen ∼ β.

Proof of Theorem 5.5.1 Lemma 5.8.7. Let pen be a polyhedral gauge on R p , X ∈ R n×p , and v ∈ col(X).

Due to uniform uniqueness, by Lemma 5.6.1 we have ker(pen) ∩ ker(X) = {0} and thus, by Lemma 5.8.7, the sequence (( β(y (r) )/r) r∈N is bounded.

Therefore, to prove that lim r→∞ β(y (r) )/r = β, it suffices to show that β is the unique accumulation point of this sequence. We extract a subsequence ( β(y q(r) )/q(r)) r∈N converging to γ ∈ R p . By (5.8.6), one may deduce that pen(γ) ≤ pen(β). Moreover, we have

Finally, γ satisfies Xγ = Xβ and pen(γ) ≤ pen(β).

Because the pattern of β is accessible, by Lemma 5.8.4 we also have pen(β) ≤ pen(γ). Then, Lemma 5.6.1 implies that γ = β. Therefore,

Lemma 5.8.9. Let pen be a polyhedral gauge on R p . Then there exists τ 0 > 0 depending on

and by the continuity of pen, one may pick τ 0 > 0 small enough such that

Proof of Theorem 5.5.1. By Lemma 5.8.9, there exists τ 0 > 0 such that for any b ∈ B ∞ (β, τ 0 ) we have ∂pen(b) ⊆ ∂pen(β). By Lemma 5.8.8, β(y (r) )/r converges to β when r tends to ∞. Consequently, we have

Consequently, for r ≥ r 0 we have

Since for any t > 0 and any x ∈ R p , we have ∂pen(x) = ∂pen(tx), one may deduce that

Consequently, the claim follows by taking τ = rτ 0 /2.

Chapter 6

Maximum likelihood estimation for discrete exponential families and random graphs Example 6.2.1. Let X = {-2, -1, 0, 1, 2} ⊂ R. Let B denote the class of all real functions on X that are of the form a + bx on {-2, -1, 0} and a + cx on {0, 1, 2} with some a, b, c ∈ R. Then {-1, 2} is a set of uniqueness for B + but {-2, 2} is not. We also observe that {-1, 2} is not a set of uniqueness for B, so the nonnegativity of functions in B + plays a role here.

Being a set of uniqueness is a monotone property in the sense that every set larger than a set of uniqueness is also of uniqueness. Furthermore, if U is a set of uniqueness for B + and A is a linear subspace of B, then U is of uniqueness for A + .

The following is a crucial definition: For U ⊂ X and ϕ ∈ B we let

Here is our characterization of the existence of MLE for discrete exponential families.

, where ψ is defined in (6.1.1). So, by (6.1.4),

To prove the other implication, we let U be a set of uniqueness for B + . By (6.1.4) for φ ∈ B,

Let C = min x∈X log µ(x). By (6.1.6), (6.1.5),

By (6.1.5) and continuity, the maximum of

The above proof is different from that of [START_REF] Bogdan | On existence of maximum likelihood estimators in exponential families[END_REF] (Theorem 2.3), [START_REF] Barndorff-Nielsen | Information and exponential families in statistical theory[END_REF] and data at hand. Having said this, for special linear spaces B one can come across interesting mathematics, as we demonstrate below. We also remark in passing that the linear problem in Corollary 6.2.1 is the Haar condition of [START_REF] Crain | Exponential models, maximum likelihood estimation, and the Haar condition[END_REF] in our setting. Quite generally, the sufficient Haar condition of Crain for the existence of MLE is in the uniqueness of a linear problem while our necessary and sufficient condition is in the uniqueness of a linear-programming problem. The latter is still computationally manageable but more subtle (and optimal); see also the last sentence in Example 6.2.1 for a difference between these two conditions in a very simple setting.

Applications

Maximization of likelihood is fundamental in estimation, model selection and testing. In many procedures it is important to know if MLE actually exists for given data x 1 , . . . , x n and the linear space of exponents B; see [START_REF] Fienberg | Maximum likelihood estimation in log-linear models[END_REF] (Introduction) for a list of such problems. [START_REF] Fienberg | Maximum likelihood estimation in log-linear models[END_REF] interpret the existence of MLE by using the geometry of the polyhedral cone spanned by the rows of a specific design matrix. This result is connected with the criterion of [START_REF] Barndorff-Nielsen | Information and exponential families in statistical theory[END_REF]. They also inquire which parameters are estimable when MLE is missing. Below we show that the notion of the set of uniqueness is useful in characterizing the existence of MLE in discrete exponential families for specific spaces B. There are two types of results we propose:

(a) conditions for the existence of MLE for a given sample, (b) probability bounds for the existence of MLE for independent identically distributed samples.

To this end let X and B be as in Section 6.1.1. Let X 1 , X 2 , . . . be i.i.d. random variables with values in X . We define the random (stopping) time

We will estimate tails of the distribution of ν uniq in terms of X , B and n. Typically we are interested in uniformly distributed X i 's: P(

In the setting of Theorem 6.2.2 we consider B = R X . We fix arbitrary µ > 0 on X , see Proof. By Theorem 6.2.2 it is enough to verify that X is the only set of uniqueness for R X + .

Obviously, X is a set of uniqueness for R X + (in fact for R X ). On the other hand, if U ⊂ X and x 0 ∈ X \ U , then 1 x 0 vanishes on U but not on X , hence U is not of uniqueness for R X + (neither it is for R X ). Example 6.3.1. Using notation of Section 6.2.1, we have U B + = U , for every U ⊂ X . Clearly, U ⊂ U B + . On the other hand, using Equation (6.2.3), one may observe that for every x / ∈ U the

Later on we give examples which use the full strength of Theorem 6.2.2 and the nonnegativity of functions in B + therein. For now we propose a probabilistic consequence of Lemma 6.3.1. Corollary 6.3.1. Let B = R X and K = |X |. Let X 1 , X 2 , . . . be independent random variables, each with uniform distribution on X . Then, for every c ∈ R,

Proof. Let ν X = inf{n ≥ 1 : {X 1 , . . . , X n } = X }. The random variable ν X yields a connection to the classical Coupon Collector Problem, see [START_REF] Erdős | On a classical problem of probability theory[END_REF], and [START_REF] Pósfai | Approximation theorems related to the coupon collector's problem[END_REF]. Namely, by [START_REF] Erdős | On a classical problem of probability theory[END_REF],

By Lemma 6.3.1, ν X = ν uniq , and the proof is complete.

We aim to cover with large probability the whole of X by a sample of suitable size depending on K.

Proof. By Lemma 6.3.1 and Corollary 6.3.1, for every c ∈ R we get lim sup

Thus lim K→∞ P (ν uniq < (1 -ε) K log K) = 0. The second part of (6.3.1) is obtained analogously. Remark 6.3.2. We summarize (6.3.1) by saying that K log K is a sharp threshold of the sample size for the existence of MLE for e(R X ) and uniform i.i.d. samples. Sharp thresholds are widely used in the theory of random graphs, see [START_REF] Erdős | On the evolution of random graphs[END_REF] (Equation 3). It is also convenient to use them here to indicate the minimal size of i.i.d. samples that guarantees the existence of MLE with high probability.

Rademacher functions

and we denote r 0 (χ) = 1. Let B k = Lin{r 0 , r 1 , . . . , r k }.

We define, as usual, the exponential family

Clearly, this is a half-cube, see (6.3.3). We will show that

We note that for j ∈ J ′ , r j is constant on the right-hand side of (6.3.4). Accordingly, the right-hand side of (6.3.4) is isomorphic to {-1, 1} |J| or to Q |J| . Now if r = k j=0 a j r j ∈ B k + and r(x 1 ) = . . . = r(x n ) = 0, then r = j∈J a j r j + c ≥ 0 on {-1, 1} |J| , where c = a 0 + j∈J ′ a j r j (x 1 ) is the sum of terms which are constant on j∈J ′ H j . In the case when J = ∅, it is obvious that {x 1 , . . . ,

However, if J ̸ = ∅, then by definition of J and Theorem 6.3.

On the other hand, we observe that for each j ∈ J ′ , 1 H c j = 0 on the sample and 1

By Theorem 6.2.5, MLE exists for e( B k ) and x 1 , . . . , x n with the measure µ := µ| X on X := j∈J ′ H j . Of course, X is isomorphic with Q |J| , if we ignore the J ′ coordinates of the points in X. In this way we may also think that µ and x 1 , . . . , x n are on Q |J| . Thus, one may calculate the supremum of the log-likelihood function for e(B k ), x 1 , . . . , x n and µ as the maximum of a log-likelihood function on Q |J| . Of course, the total mass of µ is a fraction of that of µ. For instance, if µ is the uniform probability weight on Q k then µ is uniform with the total mass 2 -|J ′ | , which adds n|J ′ | log 2 to the log-likelihood that would be obtained for Q |J| with the uniform probability weight, see, e.g., (6.1.2).

Here is a probabilistic application of Theorem 6.3.3. Corollary 6.3.4. Let k ∈ N and X 1 , X 2 , . . . , X n be independent random variables, each with uniform distribution on Q k . Then, P MLE exists for e(B k ) and X 1 , . . . , X n = 1 -

Proof. We have P(X i = x) = 2 -k for all x ∈ Q k and i = 1, . . . , n. We let R ij = r j (X i ) for i = 1, . . . , n and j = 1, . . . , k. Thus, P(R ij = 1) = P(R ij = -1) = 1

Applying the Bernoulli inequality finishes the proof.

Random graphs

In this section we focus on random graphs. Their various applications can be found in [START_REF] Rinaldo | On the geometry of discrete exponential families with application to exponential random graph models[END_REF], [START_REF] Schweinberger | Exponential-family models of random graphs: Inference in finite-, super-, and infinite population scenarios[END_REF] and [START_REF] Mukherjee | Detection thresholds for the β-model on sparse graphs[END_REF]. What is important for us, many such models are indeed discrete exponential families. As usual, maximum likelihood can be used to select a suitable graph model within the exponential family, see, e.g., [START_REF] Pitman | Some basic theory for statistical inference[END_REF] (Chapter 1 and 8) and [START_REF] Bezáková | Graph model selection using maximum likelihood[END_REF]. In this section we characterize the existence of MLE in such context. The theory of random graphs started with probabilistic proofs of the existence or nonexistence of specific graphs by Erdős, see, e.g., [START_REF] Bollobás | Modern graph theory[END_REF]. Asymptotic properties of random graphs were developed in the seminal papers of [START_REF] Erdős | On random graphs[END_REF][START_REF] Erdős | On the evolution of random graphs[END_REF] and [START_REF] Gilbert | Random graphs[END_REF]. [START_REF] Rinaldo | On the geometry of discrete exponential families with application to exponential random graph models[END_REF] discuss geometric interpretations of the existence of MLE for discrete exponential families with applications to random graphs and social networks. [START_REF] Chatterjee | Estimating and understanding exponential random graph models[END_REF] give normalizing constants that are crucial for the computation of MLE for exponential random graph models. Furthermore, they include examples when MLE fails to exist. The same authors together with Sly discuss in [START_REF] Chatterjee | Random graphs with a given degree sequence[END_REF] the asymptotic probability of the existence and uniqueness of MLE for the β-model of graphs. This allows to connect the β-model with a random uniform model of graphs with a given degree sequence, which is then explored using graphons (graph limits, see [START_REF] Lovász | Limits of dense graph sequences[END_REF]). They also present an algorithm for the computation of MLE in the β-model. [START_REF] Perry | Null models for network data[END_REF] put nonasymptotic conditions for the existence of MLE in various random graph models parameterized by vertex-specific parameters. [START_REF] Rinaldo | Maximum likelihood estimation in the βmodel[END_REF] characterize the existence of MLE for β-models. They interpret the Barndorff-Nielsen's criterion using the geometry of multidimensional polytopes of vertex-degree sequences, see also [START_REF] Fienberg | Maximum likelihood estimation in log-linear models[END_REF]. [START_REF] Wang | Approximating faces of marginal polytopes in discrete hierarchical models[END_REF] transfer the criterion into discrete hierarchical models, using the notion of simplicial complices. These models include, e.g., graphical models and Ising models. Wang, Rauh and Massam also improve the approximation of the set of estimable parameters in the case of the nonexistence of MLE, which is discussed in the setting of marginal polytopes.

Let us start with the notation. Graph is a pair G = (V, E), where V = {1, . . . , N }, N ∈ N, is the set of nodes and E is the set of edges, i.e.,

We only consider simple undirected graphs (containing no loops or multiple edges). Let m = |E|.

If m = N 2 , then the graph is called complete and is denoted as K N . On the other hand, the empty graph (with m = 0) is denoted as K N . For graphs G = (V, E 1 ) and H = (V, E 2 ) we let, as usual,

Let G N be the family of all the graphs with N nodes, i.e., with V = {1, . . . , N }. By a random graph we understand a random variable G with values in G N . The families of distributions of such random variables are called random graph models. We focus on the exponential model of random graphs G N,c defined as follows.

For 1 ≤ r < s ≤ N and G ∈ G N , we let

We define χ r,s : G N → {-1, 1} by χ r,s (G) = 1 -21 G (r, s). We consider the linear space

2 ) be a corresponding vector of coefficients. Following the setting of Section 6.1.1 we let µ(G) = 1 for each G ∈ G N (but see Remark 6.2.3) and consider the exponential family

where

for G ∈ G N , see also (6.1.3). As usual, for

) and let G be a random graph with distribution G N,c . Let 1 ≤ r < s ≤ N . Then the probability of the appearance of the edge (r, s) in G equals p r,s = e cr,s 1 + e cr,s . (6.4.

2)

The result is well known but for convenience a proof is given in Appendix 6.6.3.

Then the appearances of edges (r 1 , s 1 ) and (r 2 , s 2 ) in G are independent events.

The proof of the result is similar to that of Lemma 6.4.1, and can be found in Appendix 6.6.4. For instance, if p r,s = p ∈ (0, 1) for every edge (r, s), then the exponential random graph with distribution G N,c is the Erdős-Rényi random graph G N,p in [START_REF] Erdős | On random graphs[END_REF][START_REF] Erdős | On the evolution of random graphs[END_REF]. The latter means that P(e ∈ E(G)) = p for every edge e ∈ V 2 , and the events e ∈ E(G) and f ∈ E(G) are independent for different edges e, f .

Proof. By Theorem 6.2.2, MLE exists if and only if {G 1 , . . . , G n } is of uniqueness for B G N + . We first prove the "only if" part of Theorem 6.4.1. Let us assume that there exists an edge (r 0 , s 0 ) / ∈ n i=1 G i . Then the function χ r 0 ,s 0 ∈ B G N + equals zero on G 1 , . . . , G n , but not on the whole G N . In addition, if there is an edge (r 0 , s 0 ) ∈ n i=1 G i , then the function

+ vanishes for G 1 , . . . , G n , but it is not equal to zero, e.g., for the graph K N . We next prove the 'if' part of the theorem. Let

It follows that k 0 ≤ (r,s)̸ =(r 0 ,s 0 ) |k r,s | and eventually we get k r 0 ,s 0 = 0, thanks to (6.4.3). Since (r 0 , s 0 ) is arbitrary, k r,s = 0 for every 1 ≤ r < s ≤ N . Then also c 0 = 0, and thus ϕ ≡ 0.

In the above random graph model it is possible to compute explicitly the probability of the existence of MLE for i.i.d. samples of graphs in G N . To this end, for 1 ≤ r < s ≤ N we fix In particular, if c = 0, then the probability of the existence of MLE for e(B G N ) equals

which is an analogue of Corollary 6.3.5. From the above results we can deduce asymptotic bounds for the i.i.d. sample size for which MLE exists with high probability. To this end we recall the classical result on p = p(N ) ∈ (0, 1) such that G from G N,p has at least one edge with high probability.

Remark 6.4.2.

[80] (Lemma 1.10) Let G N,p(N ) be a random graph with distribution G N,p(N ) .

Then lim

The above may be summarized by saying that N -2 is a threshold for the probability p such that G with distribution G N,p has at least one edge. For more information on threshold functions in the theory of random graphs see, e.g., [START_REF] Frieze | Introduction to random graphs[END_REF]. In particular, a sharp threshold (mentioned previously) is a threshold but the converse is not true in general. Proof. According to Lemma 6.4.3, the probability of the existence of MLE for e(B G N ) and G 1 , . . . , G n equals

We define the function

Clearly, f (x) = f (1 -x) and for w ≥ 2 we have f increasing when 0 < x < 1 2 and decreasing when 1 2 < x < 1. Using (6.4.5) we can bound P MLE from above by

Applying Corollary 6.3.4 and the equality in (6.3.5) for k = N 2 , we observe that for every b ∈ R

We consider the sample size n = n(N ) (depending on N ). We will prove that if log N/n → 0 as N → ∞, then P MLE → 1. To this end we bound P MLE from below by

where c max = max 1≤r<s≤N |c r,s | and p max = e cmax /(1 + e cmax ).

Take n independent Erdős-Rényi random graphs H 1 , . . . , H n with distribution G N,pmax . Then the probability of the existence of MLE for e(B G N ) and for H 1 , . . . , H n equals exactly P SMALL .

Note that intersection and union of the graphs are also Erdős-Rényi random graphs, namely

where

From Remark 6.4.2, with high probability we have n i=1

By definition, c max > 0, so p max > q max . In order to get

then the above condition is satisfied. Therefore log N is a threshold of the sample size for existence of MLE for e(B G N ) and independent G 1 , . . . , G n from G N,c .

Applications to Walsh functions

We return to Rademacher functions to discuss the spaces spanned by their products. Let k ∈ N, 1 ≤ q ≤ k, and B k q = Lin {w S : S ⊂ {1, . . . , k} and |S| ≤ q} , where

each q-subcube of Q k with q ≥ 1 can be covered by disjoint pairs {e, o} as above. Therefore, the functions 1 {e,o} ∈ B k k-1 with e ∈ E and o ∈ O span the linear space

Indeed, if f = 1 {e,o} with e ∈ E and o ∈ O, then the equality is true because both sides of (6.5.4) are equal to 1. Since such functions span B k k-1 it follows that (6.5.4) is true for every

vanishes on E, then the sum over O also equals zero, hence f ≡ 0, and the same conclusion holds if we assume that f = 0 on O. Thus U is the set of uniqueness if O ⊂ U or E ⊂ U .

Remark 6.5.2. A naïve extension of Corollary 6.3.3 fails for e(B k

2 ), if we try to replace the half-cubes with (k -2)-subcubes, that is, quarter-cubes. This is seen from Theorem 6.5.1 for k = 3. Indeed, the set

2 ) + , as follows from (6.5.1) with e = (-1, -1, -1) and o = (1, 1, 1), even though the set has nonmpty intersection with each quarter-cube.

We will briefly treat the case of e(B k k ), as follows. ) because for e(B k 1 ) the sharp threshold, and so the threshold, equal log 2 k, by Corollary 6.3.6. Remark 6.5.3. Let 1 ≤ q 1 ≤ q 2 ≤ k. Then every set U of uniqueness for (B k q 2 ) + is of uniqueness for (B k q 1 ) + , because (B k q 1 ) + ⊂ (B k q 2 ) + .

A characterization of the existence of MLE for e(B k q ) for arbitrary q, even for q = 2, turned out to be difficult. Accordingly, we do not give a sharp threshold for the size of the uniform i.i.d. sample needed for the existence of MLE for e(B k q ). However, the case of e(B k k-q ) seems a little easier in the sense that we are able to give the less precise threshold for the existence of MLE for e(B k k-q ). Moreover, for each fixed q the threshold for e(B k k-q ) is the same as for e(B k k ), namely k2 k as k → ∞. Lemma 6.5.2. Fix q ∈ N. Then k2 k is a threshold of the sample size for the existence of MLE for e(B k k-q ) and i.i.d. sample uniform on Q k .

Proof. If lim k→∞ n(k)/(k2 k ) = ∞, then by Remark 6.5.3 and Corollary 6.5.2, for k → ∞ we get

as needed. On the other hand, every set U of uniqueness for (B k k-q ) + must intersect with every subcube defined by fixing last k -q coordinates, because each q-subcube is the support of a function in (B k k-q ) + , to wit, of its indicator. There are 2 k-q such q-subcubes, each of which we can suggestively denote by ( * , . . . , * , ε q+1 , . . . , ε k ), where ε q+1 , . . . , ε k = ±1. Observe that the family of the above subcubes is a partition of Q k . We consider each q-subcube as a coupon in the Coupon Collector Problem. If a sample point falls into the q-subcube, we consider the coupon as collected. The probability of collecting a given coupon is 2 q-k . Therefore, if

as needed. Let p = e(ϕ 0 ), p = e(ϕ 1 ) ∈ e(B) and p ̸ = p, so that ϕ 1 -ϕ 0 ̸ = const. Let ϕ t = ϕ 0 + t(ϕ 1 -ϕ 0 ), p t = e(ϕ t ) for t ∈ R and l(t) = l pt (x 1 , . . . , x n ). We claim that l is strictly concave, that is l ′′ < 0. Indeed, since ϕ t = ϕ 0 + tϕ 1 is a linear function, by (6.1.4) we get

Appendix

Let X be a random variable with values in X such that P(X = x) = p(x)µ(x). As usual, for every f :

Clearly, (log Z(ϕ t )) ′ = Z(ϕt) ′ Z(ϕt) and (log Z(ϕ t )) ′′ = Z(ϕt) ′′ Z(ϕt) -Z(ϕt) ′ Z(ϕt)

2

. Hence, thanks to (6.1.1),

Thus,

and so d 2

since ϕ 1 -ϕ 0 is not constant. Hence, l is strictly concave, in particular l(1/2) > (l(0) + l( 1))/2. If sup p∈e(B) L p (x 1 , . . . , x n ) = L p (x 1 , . . . , x n ) = L p (x 1 , . . . , x n ), then l(1/2) > sup p∈e(B) l p (x 1 , . . . , x n ), which is absurd; thus at most one of p and p can be the MLE.

. (6.6.3) Note that

where

Therefore

= e cr,sχr,s(G) e C(G) .

Clearly, c r,s χ r,s (G) is c r,s if (r, s) ∈ E(G) and it is 0 if (r, s) / ∈ E(G). Thus, (6.6.3) equals

Let S be the graph with only one edge (r, s). The map G → G \ S is a bijection between the graphs with the edge (r, s) and graphs without (r, s). In addition, C(G) = C(G \ S), and so we get (6.4.2).

Proof of Lemma 6.4.2

By (6.4.1), each G ∈ G N appears in G N,c with probability p c (G) = e ϕc(G)-ψ(ϕc) . Then,

G∈G N e ϕc (G) .

As in the proof of Lemma 6.4.1, we observe that

Thus,

= e cr 1 ,s 1 χr 1 ,s 1 (G) e cr 2 ,s 2 χr 2 ,s 2 (G) e C(G) .

Let S 1 and S 2 be the graphs with only one edge, (r 1 , s 1 ) and (r 2 , s 2 ), respectively. Let

Proof. The property with the minima is equivalent to the one with the maxima because B is closed upon multiplication by -1 and because max(-ϕ) = -min ϕ. Definition 6.6.1. We say that U ⊂ X propagates extrema for B if ϕ = inf X ϕ on U implies that ϕ = inf X ϕ on X for every ϕ ∈ B.

Due to Lemma 6.6.2, the property could be equivalently stated using maxima.

Lemma 6.6.3. A nonempty U ⊂ X propagates extrema for B if and only if U is of uniqueness for B ′ + .

Proof. Assume that U is of uniqueness for B ′ + . Let ϕ ∈ B and ϕ = min X ϕ on U . Then φ = ϕ -min X ϕ ∈ B ′ + and φ = 0 on U , so φ = 0 on X and ϕ = min X ϕ on X . It follows that U propagates extrema for B. Conversely, assume that U propagates extrema for B. Let ϕ ∈ B.

Then ϕ = φ + c for some φ ∈ B and c ∈ R. If ϕ ≥ 0 and ϕ = 0 on U , then φ = min X φ = -c on U , hence φ = -c on X , and so ϕ = 0 on X . Thus, U is of uniqueness for B ′ + .

Theorem The next lemma hinges on the trivial observation that if the sample mean equals the minimum, then the sample is constant. Lemma 6.6.4. {x 1 , . . . , x n } propagates extrema for B if and only if for every ϕ ∈ B, min X ϕ < max X ϕ implies min X ϕ < φ < max X ϕ.

Proof. Let {x 1 , . . . , x n } propagate extrema for B. If min X ϕ = φ, then ϕ = min X ϕ on {x 1 , . . . , x n }, hence ϕ = min X ϕ on X and so min X ϕ = max X ϕ. A similar argument works if φ = max X ϕ; see also Lemma 6.6.2. Conversely, if {x 1 , . . . , x n } does not propagate extrema for B then there is ϕ ∈ B such that ϕ = min X ϕ on {x 1 , . . . , x n }, but max X ϕ > min X ϕ. Then min X ϕ = φ < max X ϕ.

Recall the setting and notation of Section 6.1.2. The following theorem was essentially proved in [START_REF] Barndorff-Nielsen | Information and exponential families in statistical theory[END_REF] (Theorem 9.13), except that it was stated for the minimal representation of exponential families. The formulation presented in Theorem 6.6.1 below was given in [START_REF] Johansen | Introduction to the theory of regular exponential families[END_REF] (Theorem 3.5), which covers the arbitrary canonical representation and does so with a more direct proof. Notably, [START_REF] Johansen | Introduction to the theory of regular exponential families[END_REF] uses the notion of relative interior of a convex set. Let C be the convex hull of S.

We say that t ∈ R d is in the relative interior of C if for every θ ∈ R d , min y∈C θ • y < max y∈C θ • y implies min y∈C θ • y < θ • t < max y∈C θ • y. Theorem 6.6.1. [START_REF] Johansen | Introduction to the theory of regular exponential families[END_REF] (Theorem 3.5.) MLE for e(B) and x 1 , ..., x n ∈ X exists, if and only if T is in the relative interior of C.

If i, j ̸ = c, then on the main diagonal we have

Proof. For proving that claim we observe that L * 1 (K n ) = L(K n ) + E 1 = nI n -J n + E 1 . Thus L * 1 (K n ) belongs to an associative algebra being the 5-dimensional matrix space spanned by I n , E 1 , J n , E 1 J n and J n E 1 . Therefore (L * 1 (K n )) -1 has to be looked for under the form 1 n I n + aE 1 + bJ n + cE 1 J n + dJ n E 1 .

Clearly (L * (K n )) -1 is a symmetric matrix, thus d = c. Solving the linear system in a, b, c gives a = 0, b = (n + 1)/n and c = -1/n.

an if a < i, j, i ̸ = j,

Proof. Similarly to 7.3.2, the proof relies on the associative algebras being 6-dimensional (for L * ex ) or 11-dimensional (for L * in ) matrix spaces. The resulting inverses of augmented Laplacians follow from solving the corresponding systems of equations.

Eigenvalues of augmented Laplacian

Below we show that if the root consists of only one vertex c, then the determinant of L * c (G) does not depend on the choice of the root vertex. Moreover, it can be proved that it is equal to the number of spanning trees of G.

Discussion

Interpretations of the inverse of the augmented Laplacian

Below we couple the obtained inverses of L * matrices with the covariance matrices of some classical examples of random walks. At first let us remind that the covariance function of the