Professor Jean-Francois Aujol

Yannick Berthoumieu

Keywords: Image super-resolution, materials science, Gaussian mixture model, generalized Gaussian mixture model, high-dimensional data, dimensionality reduction, deep learning, generative adversarial network (GAN) alternating linearized minimization . . . . . . . . . Image super-resolution, materials science, Gaussian mixture model, generalized Gaussian mixture model, high-dimensional data, dimensionality reduction, deep learning, generative adversarial network (GAN)

I am very grateful to them for their support and very nice advice about

ii colleagues in the IMB and IMS laboratory for their help and support. I want to thank to Dr. Yiye Jiang, PhD student Paul Freulon, Dr. Nicoletta Prencipe, Dr. Anaïs Gastineau, Dr. Sebastián Tapia García in IMB. Especially Paul and Yiye who have accompanied and shared with me a lot during these 3 years.

Moreover, I wish to express my special thanks to Professor Simon MASNOU for providing me the valuable master internship at Institute Camille Jordan, University Lyon 1. Thanks to him, I have learned about the background knowledge of image processing to prepare for my PhD thesis. Secondly, I would like to thank Professor Clément MARTEAU who was very supportive of my studies and helped me get used to the new learning environment when I first arrived at the University of Lyon for my Master's program. Here, I would also like to thank my friends at the University of Lyon, especially doctoral student Jui-Ting Lu. She did not hesitate to help me discover our university and the city of Lyon with me and to help me integrate with my classmates.

Next, I wish to express my gratitude to the FRENCH-VIETNAM Master 2 in applied Mathematics program (PUF) of University of Science, Vietnam National University Ho Chi Minh City and the French Universities. Thanks to this program, I have access to the research career as well as the opportunity to find master and PhD scholarships in France. Moreover, I wish to express my special thanks to Professor Minh-Duc DUONG, Professor François James, Professor Mounir Haddou, Dr. Thanh-Hai ONG, Associate Professor Thanh-Nhan NGUYEN, Dr. Tri-Dung NGUYEN for their important academic lectures during my undergraduate and graduate programs. I also would like to thank to my Vietnamese friends Dr. Tan-Binh PHAN, Dr. Van Finally, I wish to express my gratitude to my family, my boyfriend and his family, who always love, care, and encourage me. I would especially like to thank my parents Mr. Dang-Hau NGUYEN and Mrs. Thi-Hoa VO, my elder sisters Dang-Buu Tram NGUYEN, Dang-Huyen-Tran NGUYEN , my brothers-in-laws Tuan-Khoa NGUYEN and Duc-Hieu NGUYEN and last but not least my niece Diep-Thao-Nguyen NGUYEN, my nephew Duc-Huy NGUYEN. They are my biggest motivation to see this thesis through to completion. I want to thank my parents for always creating the best possible conditions for my high school studies and for supporting my passion for math by allowing me to continue studying it at the university level. I believe they're also pleased with my achievements since completing my thesis. As a belated but heartfelt gesture, I would like to dedicate this thesis as a final gift to my mother. I miss you and I love you.

Bordeaux, January 17, 2023 Dang-Phuong-Lan NGUYEN Super-résolution multi-échelle d'images 3D en sciences des matériaux Résumé : Les développements récents des techniques d'imagerie et de l'analyse computationnelle modifient profondément la manière dont les sciences des matériaux sont abordées. L'image des matériaux est passée d'une résolution à l'échelle microscopique à une véritable échelle nanométrique pour analyser les défauts et les détails aux interfaces des matériaux. Ainsi, cette thèse traite du problème de la super-résolution (SR) afin de reconstruire les images de matériaux en haute résolution, par exemple à l'échelle nanométrique. Pour atteindre ce but, cette thèse explore plusieurs approches de la SR pour les images de matériaux.

Les algorithmes de super-résolution d'image unique (SISR) basés sur les patchs ont été remarqués et largement utilisés au cours de la dernière décennie. Récemment, les modèles de mélange gaussien généralisé (GGMM) se sont révélés être un outil approprié pour de nombreux problèmes de traitement d'images en raison de la flexibilité de leur paramètre de forme. Dans un premier temps, nous proposons d'utiliser un GGMM conjoint appris à partir de vecteurs concaténés de patchs d'entraînement à haute et basse résolution pour réaliser une image en super-résolution basée sur la technique de minimum mean square error (MMSE).

Malheureusement, la dimension des échantillons concaténés est très élevée ce qui entraîne une forte complexité de calcul des méthodes d'apprentissage des modèles de mélange tels que le modèle de mélange gaussien et le GGMM. Afin de limiter cette compléxité, nous proposons une extension de ces deux modèles intégrant une réduction de la dimensionnalité des données dans chaque composante du modèle par analyse en composantes principales. Ces modèles sont appelés respectivement PCA-GMM et PCA-GGMM. Pour apprendre les paramètres (de faible dimension) du modèle de mélange, nous proposons deux algorithmes EM différents dont l'étape M nécessite la résolution de problèmes d'optimisation sous contrainte. Ensuite, nous appliquons nos PCA-GMM et PCA-GGMM pour la super-résolution d'images de matériaux 2D et 3D en nous basant sur la méthode MMSE pour le modèle GGMM.

Sur le plan applicatif, l'étude des données de matériaux devient difficile car les images HR et LR possèdent des niveaux de contraste différents. Dans notre dernière contribution, nous étudions une approche d'apprentissage profond considérant le problème du changement de contraste dans les images de matériaux. En effet, nous proposons un réseaux antagonistes génératif (GAN) composé de deux générateurs, chacun répondant à une tâche différente. Le premier générateur traite le problème du changement de contraste, et le second se concentre sur la reconstruction des hautes fréquences du SR.

Toutes les méthodes proposées conduisent à des résultats convaincants, tant quantitatifs que visuels. En particulier, les résultats numériques des méthodes de réduction de la dimensionnalité confirment une influence modérée de la réduction de la taille sur les résultats globaux de la SR.

Mots-clés : Super-résolution, science des matériaux, les modèles de mélange Gaussien, les modèles de mélange Gaussien généralisé, réduction de la dimensionnalité, apprentissage profond, réseaux antagonistes génératif (GAN).

?listfigurename? 5.2 Reconstructions of 2D low resolution images. The columns from left to right correspond to the following materials images: Diamond with q = 2, Diamond with q = 4, FS with q = 2, FS with q = 4. ?listtablename?

4.1 PSNRs of the reconstructions of synthetically downsampled 2D images using either MMSE and EPLL approaches for GMM and LMM, GGMM with different shape parameter β. The magnification factor is q = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.2 PSNRs of the reconstructions of material 2D images with contrast change problem using either MMSE estimator and EPLL approach for GMM, LMM, and GGMM. The magnification factor is q = 2. . . [START_REF] Huang | Weinberer Densely connected convolutional networks[END_REF] 5.1 PSNRs of the reconstructions of artificially downsampled 2D images using either bicubic interpolation, a GMM, PCA-GMM for different choices of d or HDDC. The magnification factor is set to q ∈ {2, 4}. PCA-GMM produces results almost as good as GMM, with a much lower dimensionality. . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.2 Average execution time (in seconds) for the E-step and M-step in the EM algorithm for estimating the parameters of the mixture models. 78 5.3 PSNRs of the reconstructions of artificially downsampled 3D images using either nearest neighbor interpolation, GMM or PCA-GMM for different choices of d. The magnification factor is set to q = 2. As in the 2D case, PCA-GMM with small d produces results almost as good as GMM, but with a much lower dimensionality. . . . . . . . . 80 5.4 Average execution time (in seconds) of the E-step and M-step in the EM algorithm for estimating the parameters of the mixture models. 81 5.5 PSNRs of the reconstructions of 2D images using MMSE method with GMM, LMM and GGMM (with learned shape parameter β) models, and the PCA with PCA-GMM, PCA-LMM, PCA-GGMM. The magnification factor is set to q = 2. . . . . . . . . . . . . . . . . 82 5.6 PSNRs of the reconstructions of real material 2D image for magnification factor is set to q = 2 using MMSE method with GMM, LMM and GGMM (with learned shape parameter β) models, and the PCA with PCA-GMM, PCA-LMM, PCA-GGMM. . . . . . . . . . . 82 5.7 PSNRs of the reconstructions of 2D images for magnification factor is set to q = 4 using MMSE method with GMM, LMM and GGMM (with learned shape parameter β) models, and the PCA with PCA-GMM, PCA-LMM, PCA-GGMM. . . . . . . . . . . . . . . . . . . . 83 5.8 PSNRs of the reconstructions of 3D images using MMSE method and PCA-GGMM with q = 2. . . Cette thèse a été rédigée à l'Institut de Mathématiques de Bordeaux (IMB), UMR 5251 et au laboratoire IMS du CNRS (Laboratoire de l'Intégration du Matériau au Système, UMR 5218) -Université de Bordeaux -Bordeaux INP). Cette thèse a été financée par l'Agence Nationale de la Recherche (ANR) dans le cadre du projet ANR-18-CE92-0050 SUPREMATIM1 (SUPerREsolution of 3d MATerials IMages). L'objectif de ce projet est de développer des méthodes de superrésolution (SR) pour des images de matériaux en 3D à partir d'une image à basse résolution (LR) et d'une image locale guidée à haute résolution (HR). Les images 3D de matériaux sont prises dans le cadre du projet MUMMERING2 (MUltiscale, Multimodal and Multidimensional imaging for EngineeRING).

Contexte

La super-résolution (SR), qui consiste à obtenir une ou plusieurs images à haute résolution (HR) à partir d'une ou plusieurs observations à basse résolution (LR), a trouvé des applications dans de nombreux problèmes du monde réel. Au cours des deux dernières décennies, un grand nombre d'articles de recherche et de livres traitant d'objectifs pratiques spécifiques ont été rédigés. Pour un très bon aperçu des résultats obtenus jusqu'en 2014, nous vous renvoyons à l'étude exhaustive [START_REF] Nasrollahi | Super-resolution: a comprehensive survey[END_REF].

En ce qui concerne les algorithmes de super-résolution image unique (SISR), la plupart des travaux sont basés sur l'interpolation ou sur les statistiques de la paire d'images HR-LR. Diverses méthodes basées sur l'interpolation ont été proposées à partir d'interpolateurs linéaires ou non linéaires incorporant des noyaux de reconstruction adaptatifs plus ou moins sophistiqués à partir de structures locales 2 ?chaptername? 1. Introduction (Français) des images telles que les bords [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF]. Récemment, compte tenu de la définition mal conditionnée du problème de SR, des approches variationnelles ou bayésiennes ont été envisagées [START_REF] Gheche | Texture reconstruction guided by the histogram of a high-resolution patch[END_REF][START_REF] Hidane | Image zoom completion[END_REF]. Les a priori appris sont exploités dans les algorithmes SR pour concevoir un mappage statistique adapté entre l'image HR et l'image LR. Des modèles d'image populaires ont été étudiés, allant de la variation totale [START_REF] Hidane | Image zoom completion[END_REF] aux décompositions de type ondelettes, en passant par les hypothèses de sparsité [START_REF] Aguerrebere | A Bayesian hyperprior approach for joint image denoising and interpolation with an application to HDR imaging[END_REF][START_REF] Hidane | Image zoom completion[END_REF]. Plus spécifiquement, dans le contexte des problèmes inverses généraux, le travail de Buades et al. [START_REF] Buades | A review of image denoising algorithms, with a new one[END_REF] a montré un intérêt dans l'utilisation des patches et de l'hypothèse d'auto-similarité pour apprendre une mise en correspondance statistique efficace entre les images observées et restaurées. Dans un travail récent, [START_REF] Gheche | Texture reconstruction guided by the histogram of a high-resolution patch[END_REF] a développé une méthode variationnelle de SR pour les images texturées qui vise à super-résoudre la texture LR en prenant explicitement en compte les patchs HR dans un schéma de régularisation non-local couplé à plusieurs a priori d'histogrammes de caractéristiques locales de l'image [START_REF] Gheche | Texture reconstruction guided by the histogram of a high-resolution patch[END_REF]. Une autre approche de la SR basée sur l'apprentissage est celle des méthodes basées sur des exemples qui exploitent la structure récurrente de l'image des patchs à l'intérieur et à travers différentes échelles [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF]. L'objectif principal de cette classe de méthodes est d'une part d'apprendre un modèle conjoint de paires de patchs HR et LR et d'autre part d'estimer un patch HR à partir d'un patch LR donné.

Zoran et Weiss [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF][START_REF] Zoran | Natural images, Gaussian mixtures and dead leaves[END_REF] ont montré que, malgré leur simplicité, les modèles de mélanges gaussiens (GMM) sont remarquablement efficaces pour modéliser les statistiques des patchs d'images. Dans les tâches de restauration d'images, ils améliorent d'autres méthodes préalables génériques comme, par exemple, le codage parcimonieux.

Récemment, Sandeep et Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] ont montré le potentiel de la modélisation paramétrique pour apprendre la cartographie statistique en SR dans le cas où le noyau de flou est inconnu. En supposant un grand ensemble de patchs de paires d'images exemples HR et LR sous le modèle SR, les auteurs ont démontré que le GMM conjoint est une proposition très compétitive pour les méthodes basées sur l'apprentissage en termes de performance SR et de charge de calcul. Ils proposent une étude comparative avec un large panel de méthodes qui tirent parti de techniques d'apprentissage bien connues consacrées à l'appariement des dictionnaires HR-LR.

Dans le contexte du traitement d'images à partir de l'extraction de caractéristiques de patchs, des travaux récents [START_REF] Deledalle | Image denoising with generalized Gaussian mixture model patch priors[END_REF] tirent parti du modèle de mélange gaussien généralisé (GGMM) pour apprendre les statistiques des patchs d'images. C. Deledalle et al. ont montré que le GGMM obtient de meilleures distributions de patchs qu'un GMM et surpasse le GMM lorsqu'il est utilisé dans le même cadre que [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF]. Divers travaux [START_REF] Allili | Wavelet modeling using finite mixtures of generalized Gaussian distributions: Application to texture discrimination and retrieval[END_REF][START_REF] Boukouvalas | A new Riemannian averaged fixed-point algorithm for MGGD parameter estimation[END_REF][START_REF] Pascal | Parameter estimation for multivariate generalized Gaussian distributions[END_REF] ont montré la possibilité de considérer des modèles à fonction de densité de probabilité, qui ont plus de flexibilité pour s'adapter à la forme des données et moins de sensibilité à la sur-adaptivité du nombre de classes que le GMM.

Motivations et objectifs

Motivation en sciences des matériaux

Les développements récents et en cours dans les techniques d'imagerie et l'analyse computationnelle modifient profondément la façon dont la science et l'ingénierie des matériaux considèrent leurs sujets de recherche. Notre projet contribue à cette direction de recherche en développant de nouvelles méthodes de SR guidées par une sous-image locale de HR des données de matériaux 3D. L'imagerie 3D est passée d'une résolution à micro-échelle à un véritable régime nano-échelle. Cela permet le passage de l'analyse statistique à l'analyse des modulations des défauts et des détails aux interfaces. Les interfaces entre les phases des matériaux sont des caractéristiques importantes et déterminantes pour les performances de la science des matériaux. Elles sont décrites comme dans la Figure 1.1. Les méthodes d'imagerie multiéchelle constituent le pont entre la compréhension à nano-échelle et les modèles de performance à micro-échelle, non seulement pour les systèmes statiques, mais aussi pour mesurer les paramètres structurels clés localement et en fonction du temps.

Le projet MUMMERING travaille sur le transport réactif dans les milieux poreux: une approche d'imagerie et de modélisation multi-échelle. Des approches d'imagerie et de modélisation multi-échelles ont été utilisées pour étudier le transport réactif dans les milieux poreux où les réactions hétérogènes sont fondamentales, ce qui signifie que les frontières solides doivent être décrites avec précision. D'autre part, la distribution spatiale globale du solide doit également être caractérisée avec précision, car elle détermine les propriétés moyennes de transport. La méthode SR d'image nous permet de compléter l'image LR globale avec des informations locales extraites d'une région donnée de l'image HR et LR. Cela permet d'affiner de manière itérative une simulation numérique des propriétés de transport dans l'espace global avec des images HR. À cette fin, notre projet (SUPREMATIM) a été construit pour explorer le modèle statistique reliant les images LR et HR en collaboration avec le groupe allemand du professeur Gabriele Steidl à la Technische Universität Berlin et ?chaptername? 1. Introduction (Français) du professeur Claudia Redenbach à la Technische Universität Kaiserslautern.

Dans le projet SUPREMATIM, nous examinons deux problèmes transitoires de la science des matériaux (défaillance des composites en aluminium et réactions électrochimiques dans les électrodes poreuses), pour lesquels l'imagerie à nanoéchelle et à micro-échelle est nécessaire. Les perspectives de développement de la SR comme méthode de propagation de l'information géométrique de nano-échelle à micro-échelle sont encore plus stimulantes ici, puisque l'imagerie à micro-échelle (tomographie aux rayons X) est non destructive, mais que l'imagerie à nano-échelle pourrait être destructive (tomographie au microscope électronique à balayage à faisceau d'ions focalisé (FIB-SEM)). En fait, la caractérisation 3D des matériaux considérés ici serait facilitée si les acquisitions HR destructives étaient limitées à des zones suffisamment petites pour maintenir la représentativité de l'échantillon, lorsque les acquisitions LR et HR sont réalisées.

Objectifs

L'objectif de cette thèse est le développement de méthodes de SR pour des images 3D uniques provenant d'images LR et HR de la même sonde de matériaux. Sur la base de nos motivations, nous avons décidé de développer de nouvelles méthodes de SR guidées par des sous-images HR locales à partir d'images LR globales.

Pour cela, le premier objectif de cette thèse est d'étendre la super-résolution d'image utilisant la méthode GMM des travaux précédents [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF][START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF] au cas GGMM. Nous avons l'intention de capturer les statistiques HR de l'image LR d'une manière plus appropriée en utilisant cette modélisation plus riche. Le deuxième objectif de cette thèse est d'étudier les techniques de réduction de la dimensionnalité qui incorporent un GMM ou un GGMM. Enfin, nous considérons les techniques d'apprentissage profond pour la super-résolution.

Résumé de nos contributions

Sur la base de nos objectifs, cette thèse est divisée en quatre contributions principales. Tout d'abord, pour aborder la SR d'images dans des images matérielles 3D guidées par des sous-images HR locales issues d'images LR globales, nous avons choisi de nous placer dans le cadre d'une approche basée sur l'exemple. Cette approche s'appuie sur des jeux de données spécifiques fournis à partir de sousimages HR locales et d'images LR correspondantes issues d'images LR globales. Elle étend la méthode SR de Sandeep et Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] au modèle de mélange gaussien généralisé. À cette fin, nous proposons un algorithme comme première contribution pour apprendre un modèle de mélange gaussien conjoint basé sur l'algorithme espérance-maximisation (EM) à partir d'une paire de patchs LR et des patchs HR correspondants provenant des données de référence. Les paramètres du GGMM sont estimés en utilisant les itérations d'un algorithme de point fixe (FP) pour l'estimation du maximum de vraisemblance pondéré dans l'algorithme EM. Cet algorithme EM incorporant des itérations FP pour le modèle GGMM est appelé 1.3. Résumé de nos contributions algorithme FP-EM. Nous reconstruisons ensuite l'image HR en utilisant l'estimateur de l'erreur quadratique moyenne minimale (MMSE) pour le modèle GGMM. Cette proposition est appelée la méthode MMSE-GGMM. En outre, l'image du matériau est passée d'une résolution à l'échelle du micron à une véritable échelle nanométrique. L'étude des données matérielles est très difficile car les images HR et LR ont des contrastes différents. De plus, la connaissance de l'opérateur SR n'est pas connue. Dans cette contribution, nous démontrons que notre méthode gère l'opérateur inconnu et est invariante au problème de changement de contraste affine.

La deuxième contribution consiste en une nouvelle approche pour la réduction de la dimension des données avec des applications en super-résolution. En effet, la réduction de dimensionnalité devient importante dans l'apprentissage de modèles statistiques lorsque la dimension des données est élevée. Tout d'abord, nous proposons un modèle de mélanges gaussiens en conjonction avec une réduction de la dimensionnalité des données dans chaque composante du modèle par analyse en composantes principales (PCA), appelée PCA-GMM. Ce modèle est généré en ajoutant le terme de minimisation de la PCA et la fonction de log-vraisemblance négative du GMM sur les points de données de dimension réduite. Ensuite, pour apprendre les paramètres (de faible dimension) du modèle de mélange, nous proposons un algorithme EM dont l'étape M nécessite la résolution de problèmes d'optimisation sous contrainte. Deuxièmement, nous appliquons notre PCA-GMM pour la SR d'images de matériaux 2D et 3D en nous basant sur l'approche de Sandeep et Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF]. Les résultats numériques confirment l'influence modérée de la réduction de la dimensionnalité sur le résultat du SR.

La troisième contribution étend la contribution précédente au modèle GGMM pour le problème de réduction de la dimensionnalité. Dans ce travail, nous proposons de combiner un GGMM avec une PCA et de dériver un algorithme EM pour estimer les paramètres du modèle qui en résulte. Le nouveau modèle est appelé modèle PCA-GGMM. Pour gérer la complexité de l'algorithme EM pour le modèle PCA-GGMM, la technique de séparation semi-quadratique est utilisée dans l'étape d'estimation du maximum de vraisemblance. Ensuite, le modèle PCA-GGMM peut être appliqué au SR en utilisant la méthode MMSE-GGMM.

Dans la dernière contribution, cette thèse propose une nouvelle méthode SR basée sur l'apprentissage profond pour les images de matériaux avec le problème de changement de contraste. Parmi tous les types de réseaux de neurones convolutifs qui existent, nous avons choisi de nous placer dans le contexte des réseaux antagonistes génératifs (GAN). En effet, ces dernières années, les GAN ont commencé à émerger pour le problème de la super-résolution, notamment suite à la méthode proposée par Ledig et al. [START_REF] Ledig | Photo-realistic single image superresolution using a generative adversarial network[END_REF]. Cependant, ils manquent encore d'efficacité lorsque les contrastes des images HR et LR changent. Dans notre première contribution, nous avons montré que le MMSE-GGMM peut être invariant avec le problème de changement de contraste sous une transformation affine. Mais cette méthode n'est pas garantie pour le cas non affine. Pour surmonter ce problème avec une fonction de transformation non affine, nous proposons un nouveau cadre GAN dans lequel le générateur G est divisé en deux sous-réseaux. Le premier sous-réseau traite du problème des changements de contraste, et le second sous-réseau se concentre sur la reconstruction des hautes fréquences du SR.

Organisation du manuscrit

Cette thèse traite du traitement de super-résolution de données matérielles en utilisant des méthodes basées sur l'apprentissage, en particulier, basées sur des modèles de mélanges gaussiens généralisés locaux et globaux. Cette thèse apporte quatre contributions à ce sujet. Notre première contribution vise à prendre en compte une nouvelle méthode de super-résolution qui utilise l'estimateur MMSE pour le modèle de mélanges gaussiens généralisés GGMM. Les deuxième et troisième contributions de cette thèse consistent à introduire deux modèles qui combinent la méthode PCA de réduction de dimensionnalité avec le modèle de mélanges gaussiens et le modèle de mélanges gaussiens généralisés. La dernière contribution est de concevoir l'architecture d'un GAN pour nos données matérielles avec les défis de changement de contraste et d'opérateur de corruption inconnu.

Dans le chapitre 3, nous donnons une vue d'ensemble des données sur les matériaux et un état-de-l'art général des méthodes existantes pour aborder et résoudre le problème de la super-résolution des images de matériaux. Les défis de la super-résolution pour notre image matérielle sont mentionnés dans ce chapitre. Il s'agit des changements de contraste, de luminosité de l'image, et des problèmes de super-résolution avec un opérateur de corruption inconnu. Il existe une grande variété de méthodes pour résoudre le problème de super-résolution d'image. En effet, il existe des approches basées sur l'interpolation, des approches basées sur la reconstruction ou des approches basées sur l'exemple. Par conséquent, ce chapitre vise à souligner les forces et les faiblesses de chaque type de méthodes.

Le chapitre 4 est consacré à la méthode de super-résolution guidée par la sousimage HR locale basée sur l'approche par exemple. Tout d'abord, nous présentons un état de l'art détaillé des approches basées sur les patchs pour le problème de la super-résolution. Il s'agit notamment de la méthode de super-résolution via le modèle de mélanges gaussiens conjoints [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] et de la méthode EPLL (Expected patch log-likelihood) pour la super-résolution [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF]. Nous terminons ce chapitre en nous concentrant sur notre première contribution concernant l'approche MMSE-GGMM du problème de la SR. Cette contribution a fait l'objet d'une publication lors de la conférence GRETSI 2022 [START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF].

Le chapitre 5 traite du défi de la haute dimension des données matérielles. Pour résoudre ce problème, nous étudions quelques solutions de pointe au problème de la réduction de la dimensionnalité. Cela inclut le modèle PCA, l'analyse probabiliste en composantes principales (PPCA) pour la distribution gaussienne, le modèle MPPCA (Mixture of Probabilistic Principal Component Analyzers) et la méthode HDDC (High-Dimensional Data Clustering). Ensuite, nos deuxième et troisième contributions sont présentées dans le contexte de leurs applications en super-résolution. La deuxième contribution a été publiée dans la revue Inverse This thesis was written at the Institute of Mathematics of Bordeaux (IMB), UMR 5251 and the IMS laboratory (Laboratoire de l'Intégration du Matériau au Système, UMR 5218) in CNRS -Université de Bordeaux -Bordeaux INP). This thesis was funded through the French Agence Nationale de la Recherche (ANR) under the reference project ANR-18-CE92-0050 SUPREMATIM 1 (SUPerREsolution of 3d MATerials IMages). The aim of this project is the development of superresolution (SR) methods for 3D materials images originating from a low-resolution (LR) image and a guided local high-resolution (HR) sub-image. The 3D materials images are taken within the MUMMERING 2 project (MUltiscale, Multimodal and Multidimensional imaging for EngineeRING).

Context

Super-resolution (SR), the process of obtaining one or more high-resolution (HR) images from one or more low-resolution (LR) observation(s) has found applications in many real-world problems. Over the past two decades a large number of research papers and books addressing specific practical purposes have been written. For a very good overview of the results until 2014 we refer to the comprehensive survey [START_REF] Nasrollahi | Super-resolution: a comprehensive survey[END_REF].

Considering single image super-resolution (SISR) algorithms, most of the works are interpolation-based or based on the statistics of the pair HR-LR images. Various interpolation-based methods have been proposed from linear or nonlinear interpolators incorporating more or less sophisticated adaptive reconstruction kernels from local image structures such as edges [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF]. Recently, taking into account 10 ?chaptername? 2. Introduction the underlying ill-conditioned definition of the SR problem, variational or Bayesian approaches have been considered [START_REF] Gheche | Texture reconstruction guided by the histogram of a high-resolution patch[END_REF][START_REF] Hidane | Image zoom completion[END_REF]. The learned priors are exploited in the SR algorithms to design an adapted statistical mapping between the HR image and the LR image. Popular image models have been studied ranging from total variation [START_REF] Hidane | Image zoom completion[END_REF], wavelet-like decompositions, or sparsity assumptions [START_REF] Aguerrebere | A Bayesian hyperprior approach for joint image denoising and interpolation with an application to HDR imaging[END_REF][START_REF] Hidane | Image zoom completion[END_REF]. More specifically, in the context of general inverse problems, the special work of Buades et al. [START_REF] Buades | A review of image denoising algorithms, with a new one[END_REF] has shown an interest in using patches and self-similarity hypothesis to learn efficient statistical mapping between observed and restored images. In a recent work, [START_REF] Gheche | Texture reconstruction guided by the histogram of a high-resolution patch[END_REF] has developed a variational SR method for textured images that aims to superresolve the LR texture by explicitly taking into account HR patches in a non-local regularization scheme coupled with several histogram priors of local image features [START_REF] Gheche | Texture reconstruction guided by the histogram of a high-resolution patch[END_REF]. Another approach to learning-based SR is example-based methods that exploit the recurrent image structure of patches within and across different scales [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF]. The main purpose of this class of methods is on the one hand to learn a joint model of HR and LR patch pairs and on the other hand to estimate a HR patch from a given input LR patch.

Zoran and Weiss [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF][START_REF] Zoran | Natural images, Gaussian mixtures and dead leaves[END_REF] showed that, despite their simplicity, Gaussian mixture model (GMM) is remarkably good at modeling the statistics of natural image patches. In image restoration tasks they improve other generic prior methods such as for instance sparse coding.

Recently, Sandeep and Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] have shown the potential of parametric modeling to learn statistical mapping in SR in the case where the blur kernel is unknown. Assuming a large patch set of HR and LR pairs of example images under the SR model, the authors demonstrated that joint GMM is a very competitive proposal for learning based methods in terms of SR performance and computational load. They propose a comparative study with a large panel of methods that take advantage of well-known learning techniques devoted to pairing HR-LR dictionaries.

In the context of image processing from patch feature extraction, recent work [START_REF] Deledalle | Image denoising with generalized Gaussian mixture model patch priors[END_REF] takes advantage of generalized Gaussian mixture model (GGMM) to learn the statistics of image patches. C. Deledalle et al. showed that the GGMM obtains better patch distributions than a GMM and outperforms the GMM when used in the same framework with [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF]. Various works [START_REF] Allili | Wavelet modeling using finite mixtures of generalized Gaussian distributions: Application to texture discrimination and retrieval[END_REF][START_REF] Boukouvalas | A new Riemannian averaged fixed-point algorithm for MGGD parameter estimation[END_REF][START_REF] Pascal | Parameter estimation for multivariate generalized Gaussian distributions[END_REF] have shown the possibility of considering probability density function models, which have more flexibility to adapt the shape of data and less sensibility to overfitting the number of classes than the GMM.

Motivations and objectives

Motivation from Materials Sciences

Recent and ongoing developments in imaging techniques and computational analysis deeply modify the way materials science and engineering consider their research subjects. Our project contributes to this direction of research by developing new SR methods guided by a local HR sub-images of 3D materials data. 3D imaging ?figurename? 2.1: 3D rendering of a fuel cell electrode with, on the left part, porosity (transparent), electronic conductor (green) and ionic conductor (yellow). On the right part the red lines represent the Triple-Phase-Boundaries where electrochemical reactions take place.

has moved from microscale resolution to true nanoscale regime. This allows the passage from statistical analysis to the analysis of the modulations of the defects and details at the interfaces. The interfaces between the phases of materials are important and determining characteristics for the performance of materials science. They are described as in Figure 2.1. Multiscale imaging methods form the bridge between the nanoscale understanding and the microscale performance models, not only for static systems, but also to measure key structural parameters locally and as a function of time.

The MUMMERING project is working on reactive transport in porous media: A multiscale imaging and modelling approach. Multiscale imaging and modeling approaches were used to investigate reactive transport in porous media where heterogeneous reactions are fundamental, meaning that the solid boundaries must be precisely described. On the other hand, the global spatial distribution of the solid must also be precisely characterized, because it determines the average transport properties. The image SR method allows us to supplement the global LR image with local information extracted from a given HR and LR image region. This allows for the iterative refinement of a numerical simulation of transport properties in global space with HR images. To this end, our project (SUPREMATIM) was built to explore the statistical model linking LR and HR images in collaboration with the German group of Professor Gabriele Steidl at Technische Universität Berlin and Professor Claudia Redenbach at Technische Universität Kaiserslautern.

In the SUPREMATIM project, we consider two transitory problems of materials science (failure of Aluminum composites and electrochemical reactions in porous electrodes), where nanoscale and microscale imaging are necessary. The perspectives of the development of SR as a method to propagate the geometrical information from the nanoscale to the microscale are even more stimulating here, since microscale imaging (X-ray tomography) is non-destructive, but nanoscale imaging could be destructive (Focused Ion Beam Scanning Electron Microscope (FIB-SEM) tomography). In fact, 3D characterization of the materials considered here would ?chaptername? 2. Introduction be facilitated if destructive HR acquisitions were limited to zones small enough to maintain the representativeness of the sample, when LR and HR are acquired.

Objectives

The aim of this thesis is the development of SR methods for single 3D images originating from LR images and HR images from the same material probe. Based on our motivations, we decided to develop new SR methods guided by local HR sub-images from global LR images.

For this, the first goal of this thesis is to extend the image super-resolution using the GMM method of previous work [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF][START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF] to the GGMM case. We intend to capture the HR statistics from the LR image in a more appropriate way by using this richer modeling. The second goal of this thesis is to study the dimensionality reduction techniques which incorporate a GMM or GGMM. Finally, we consider deep learning techniques for super-resolution.

Summary of our contributions

Based on our objectives, this thesis is divided into four main contributions. First of all, to address image SR in 3D material images guided by local HR sub-images from global LR images, we have chosen to place ourselves in the framework of an example-based approach. This approach relies on specific datasets provided from local HR sub-images and corresponding LR images from global LR images. It extends the SR method of Sandeep and Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] to the generalized Gaussian mixture model. To this end, we propose an algorithm as a first contribution to learn a joint GGMM based on the Expectation-Maximization (EM) algorithm from a pair of LR patches and the corresponding HR patches from the reference data. The parameters of the GGMM are estimated by using the Fixed-Point (FP) iterations for the weighted maximum likelihood estimation in the EM algorithm. This EM algorithm incorporated with FP iterations for the GGMM model is called the Fixed-Point Expectation-Maximization (FP-EM) algorithm. We then reconstruct the HR image using the Minimum mean square estimator (MMSE) estimator for the GGMM model. This proposal is called the MMSE estimator for joint GGMM (MMSE-GGMM) method. Furthermore, the material image has moved from micronscale resolution to true nanoscale. Studying the material data is very challenging because the HR and LR images have different contrasts. In addition, the knowledge of the SR operator is not known. In this contribution, we demonstrate that our method handles the unknown operator and is invariant to the affine contrast change problem.

The second contribution consists of a new approach for data dimensionality reduction with applications in super-resolution. In fact, dimensionality reduction becomes important in learning statistical models when the dimension of the data is high. First, we propose a Gaussian Mixture Model in conjunction with a reduction of the dimensionality of the data in each component of the model by Principal Component Analysis (PCA), called PCA -Gaussian mixture model (PCA-GMM). This model is generated by adding the minimization term of the PCA and the negative log-likelihood function of the GMM on the reduced-dimensionality data points. Then, to learn the (low-dimensional) parameters of the mixture model, we propose an expectation-maximization (EM) algorithm whose M-step requires the solution of constrained optimization problems. Second, we apply our PCA-GMM for the SR of 2D and 3D material images based on the approach of Sandeep and Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF]. Numerical results confirm the moderate influence of the dimensionality reduction on the overall SR result.

The third contribution extends the previous contribution to the GGMM model for the dimensionality reduction problem. In this work, we propose to combine a GGMM with a PCA and derive an EM algorithm for estimating the parameters of the arising model. The new model is called the PCA -generalized Gaussian mixture model (PCA-GGMM) model. To handle the complexity of the EM algorithm for the PCA-GGMM model, the half-quadratic splitting technique is used in the maximum likelihood estimation step. Then the PCA-GGMM model can be applied to SR by using the MMSE-GGMM SR method.

In the last contribution, this thesis proposes a new SR method based on deep learning for material images with the contrast change problem. Among all the types of convolutional neural networks that exist, we have chosen to place ourselves in a framework of Generative adversarial network (GAN). Indeed, in recent years, GAN has begun to emerge for the super-resolution problem, in particular following the method proposed by Ledig et al. [START_REF] Ledig | Photo-realistic single image superresolution using a generative adversarial network[END_REF]. However, they still lack effectiveness when the contrasts of the HR and LR images change. In our first contribution, we showed that MMSE-GGMM can be invariant with the contrast change problem under an affine transformation. But this method is not guaranteed for the non-affine case.

To overcome this problem with a non-affine transformation function, we propose a new GAN framework in which the generator G is divided into two sub-networks. The first sub-network deals with the problem of contrast changes, and the second sub-network focuses on the reconstruction of the high frequencies of the SR.

Organisation of the manuscript

This thesis discusses super-resolution processing of material data using learningbased methods, in particular, based on local and global generalized Gaussian mixture models. This thesis provides four contributions to the topic. Our first contribution aims to take into account a new method for super-resolution that uses the MMSE estimator for the generalized Gaussian mixture model. The second and third contributions in this thesis are to introduce two models that combine the PCA method of dimensionality reduction with the Gaussian mixture model and the generalized Gaussian mixture model. The last one is to design the architecture of a generative adversarial network (GAN) for our material data with the contrast change and unknown corruption operator challenges.

In Chapter 3, we give an overview of material data and a general state-of-theart on existing methods to address and solve the image super-resolution problem for material image. The challenges of super-resolution for our material image are mentioned in this chapter. These are the changes in contrast, brightness of image, and super-resolution problems with unknown corruption operator. There are a wide variety of methods to address the image super-resolution problem. Indeed, there are interpolation-based approach, reconstruction-based approach, and example-based approach. Therefore, this chapter aims to outline the strengths and weaknesses of each type of method.

Chapter 4 is devoted to the super-resolution method guided by local HR subimage based on the example-based approach. First, we present a detailed stateof-the-art of patch-based approaches for super-resolution problem. It includes the super-resolution method via the joint Gaussian mixture model [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] and the Expected patch log-likelihood (EPLL) method for super-resolution [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF]. We finish this chapter by focusing on our first contribution concerning the MMSE-GGMM approach to the SR problem. This contribution was the subject of a publication at the GRETSI conference 2022 [START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF].

Chapter 5 deals with the high dimension challenge of material data. To address this problem, we study some state-of-the-art solutions of the dimensionality reduction problem. This includes the PCA model, the Probabilistic Principal Component Analysis (PPCA) for Gaussian distribution, the Mixture of Probabilistic Principal Component Analyzers (MPPCA) model, and the High-Dimensional Data Clustering (HDDC) method. Then, our second and third contributions are presented in the context of their applications in super-resolution. The second contribution was published in the Inverse Problems in Imaging journal [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] in 2022. The third contribution was submitted to the Inverse Problems in Imaging journal [START_REF] Nguyen | Image superresolution with PCA Reduced generalized Gaussian mixture models[END_REF] in 2022.

Then, Chapter 6 is oriented towards methods based on generative adversarial network (GAN) for the contrast change problem of our material images. This chapter begins with a brief introduction to the Convolutional neural network (CNN) and GAN for the SR problem. Then the last contribution to the GAN network with two sub-networks in the generator is mentioned.

Chapter 7 concludes this thesis and proposes avenues for improvement and perspectives for the SR task for the material images.

?chaptername? 3 Super-resolution for 3D images from materials science In this chapter, we present the fundamentals of SR, by providing a general definition in Section 3.1. Section 3.2 discusses our materials data of this thesis. First, we present the material images on which most of the work in this thesis is based. After that, some challenges of super-resolution for materials data will be presented. Finally, we describe some main categories of approaches for SISR, and the SR imaging model used throughout this work.

Super-resolution definition

In this section, we provide some brief definitions of the digital image and related definitions of super-resolution.

Digital image A digital image can be considered as a discrete representation of data that contains spatial (layout) and intensity (color) information. The discrete two-dimensional (2D) digital image I is represented by a 2D array. Each element of the array is called pixel, derived from the term "picture element". Thus, a M × N digital image I looks like this

I =      I(1, 1) I(1, 2) • • • I(1, N ) I(2, 1) I(2, 2) • • • I(2, N ) . . . . . . . . . . . . I(M, 1) I(M, 2) • • • I(M, N )      (3.1.1)
?chaptername? 3. Super-resolution for 3D images from materials science where I(m, n) represents the fixed pixel (m, n) of image I (m = 1, 2, ..., M and n = 1, 2, ..., N ). Image quality is a complex concept, largely subjective, and highly application-dependent. Basically, a good quality image is an image that has the following properties:

• it has a high resolution,

• it is not noisy,

• it has good contrast.

We discuss these definitions in the following of this subsection.

Image resolution In general terms, image resolution means the amount of detail that an image holds. The resolution of an image can be described in many different ways. According to the particular aspect taken into account, it can be mentioned as: pixel count, spatial resolution, spectral resolution, temporal resolution, radiometric resolution, etc. In particular, spatial resolution refers to the level of visual details discernible in an image. Higher resolutions mean that there are more pixels per inch (PPI), resulting in more pixel information and creating a high-quality, crisp image. As for digital images, where the pixel is the base unit used, there is a connection between spatial resolution and the total number of pixels. In practice, we often take into account simply the total counts of pixels, horizontally and vertically, which define what is more precisely called pixel resolution, and serves as upper bounds on spatial resolution. In this manuscript, the word resolution is intended as a spatial resolution. The resolution of a 2D image is commonly quoted as C × R (column by row), e.g. 640 × 480, 512 × 512, 2512 × 2512, etc. An example of quality of "cameraman image" with different resolutions is given in It means that for a given low-resolution image input, our task is to estimate the corresponding high-resolution image output and it must also provide an accurate image signal without distortion.

?chaptername? 3. Super-resolution for 3D images from materials science

Super-resolution for materials data

As we discussed in Chapter 2, our motivation is to develop new super-resolution methods for data from 3D material images. In the frame of the ITN MUMMERING project, a series of multiscale 3D images has been acquired by synchrotron microtomography at the TOmographic Microscopy and Coherent rAdiology experimentTs (TOMCAT1 ) beamline of the Swiss Light Source (SLS2 ) at the Paul Scherrer Institute in Switzerland.

The series of multi-scale 3D images of two different samples were acquired following a protocol conceived for testing and validating the SR process by the following steps:

-Firstly, we selected samples with micro-geometries of different levels of complexity.

-Secondly, we acquired four tomograms for each sample for three magnifications 2, 4, and 10 following the scheme given Figure 3.4. Two different regions were scanned at a magnification of 10. The first part is used for training the SR model and the other for comparing the real and reconstructed micro-geometry.

Materials of two samples were selected to provide 3D images having diverse levels of complexity:

-The first one is a sample of Fontainebleau sandstone ("FS"), a natural rock rather homogeneous and commonly used in the oil industry for flow experiments.

-The second is a composite ("SiC Diamonds") obtained by microwave sintering of silicon and diamonds, see [START_REF] Vaucher | On-line tools for microscopic and macroscopic monitoring of microwave processing[END_REF].

Each sample has been taken with different Magnification (MAG) of 2x, 4x, and 10x. The 3D data with 2x and 4x magnification were collected for the entire sample with 1600×1600×962 and 2560×2560×2160 voxels, respectively. With magnification 10x, two regions of sample were obtained by perform local microtomography resulting 3D volumes with 2560×2560×2160 voxels. Figure 3.4 illustrates the multiscale imaging experiment with cross sections of the Fontainebleau sandstone sample. The two FS images with MAG of 10x were taken from the yellow and red circles of the FS image with MAG of 4x. This means that on 10x 3D data we only have 2 private local data, not the whole sample. We assume that 2x 3D data is a 3D LR image, and 10x 3D local parts of the data are 3D HR images. Thus we have 2 parts of HR image, not the entire HR image. This leads to the idea that we study an approach to perform super-resolution image guided by HR local subimage of 3D materials data. From the HR local subimage, the LR local subimage can be retrieved. These HR and LR sub-images are to be used as reference images for training step of super-resolution.

By extracting these subimages into small patches, we obtain a set of HR patches and the corresponding LR patches as the dictionaries. From these points of view, this thesis focuses on the research and development of example-based SR methods. In this thesis, we propose new super-resolution methods based on Generalized Gaussian Mixture Models. These methods will be presented in Chapter 4 and Chapter 5. In this section, we investigate some of the challenges and solutions of superresolution 3D materials data.

Detecting local HR and LR images via Scale-invariant feature transform (SIFT) algorithm

In materials science, 3D data is acquired using a synchrotron microtomography for different magnifications at different times. Therefore, the positions of the subject are ?chaptername? 3. Super-resolution for 3D images from materials science changed between the HR and LR images. This is illustrated in detail in Figure 3.5.

Assume that we consider a cross section with 2x MAG as a 2D LR image and a cross section with 4x MAG as a 2D HR image. In Figure 3.5, the first column shows the HR images of FS and Sic-Diamond, and the second column displays the LR images, which we acquired from synchrotron microtomography. One can easily notice that the LR image has more black areas than the HR image in the bounder of the sample. Additionally, the 10x MAG data is obtained with 2 separate local parts, not the whole data as shown in Figure 3.4. These two arguments lead us to the important task before image super-resolution, which is to find the position corresponding to the HR image in the LR image in case of 4x MAG or to detect the HR parts in the LR image in case of 10x MAG. To detect HR and LR images, we refer to the Scale-invariant feature transform (SIFT) algorithm [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF].

?figurename? 3.5: Top: Material "FS", bottom: Material "SiC Diamonds". First column: Ground-truth HR image, second column: Input LR image, third column: Match LR images. The red lines in the second column are the boundaries of the LR image that correspond to the HR image.

For more than a decade, the SIFT [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF] has arguably been the most popular keypoint detection and matching method. The SIFT algorithm proposed by David Lowe consists of a keypoint detection which is independent towards image rotation, scale change, affine transformation, intensity variation, and viewpoint change. Using the SIFT algorithm, we determine all the match points between the HR and LR images as shown in Figure 3.6. Then, the boundary of the LR image will be found by selecting the minimum and maximum coordinates of the keypoints. Finally, we obtain the 2D LR image which matches with the 2D HR image. The matched LR image is given in the third column in Figure 3.5.

?figurename? 3.6: Matched points between the HR and LR image by using SIFT

Modeling contrast change parameters

In practice, the high-resolution image and its low-resolution correspondence were acquired under different conditions. Therefore, they may have different brightness and contrast. In fact, the LR and HR images of the FS and sic Diamond data in Notably, there exists a celebrated approach which has been explored, for instance, in midway image equalization [START_REF] Delon | Midway image equalization[END_REF][START_REF] Guillemot | Implementation of the midway image equalization[END_REF]. The idea of mid-way equalization [START_REF] Delon | Midway image equalization[END_REF] is to give a pair of images the same intermediate histogram. However, in our project, our aim is to reconstruct the HR image based on the guided HR part of the ?chaptername? 3. Super-resolution for 3D images from materials science LR image. Therefore, the basic idea is to determine an affine transformation such that the LR image applied to this transformation has the same contrast as the HR image.

Let C (t) = αt + β with α, β ∈ R, α > 0 be an affine contrast change function such that Ỹ = C (Y ) has the same contrast as the HR image X, where Y denotes the LR image. In the case of data with two gray-level regions, such as our material data, we can find the maximum values in the two regions of the histogram. We denote these two values of HR image by m H , M H , and LR image by m L , M L respectively. Thanks to the property of the affine function C, we get the following equation:

m H = αm L + β M H = αM L + β.
The affine contrast change function can be easily found by solving the above system of linear equations. In this manuscript, we propose a method to reconstruct the HR image without estimating the parameters α, β of the affine contrast change function C as a pre-processing step. This statement is discussed in chapter 4.

Modeling super-resolution operator

The last challenge of super-resolution materials image processing is the SISR with unknown corruption, the major challenge in the SR community. The SISR aims to reconstruct an HR image from a single LR input image. In general, the LR image is generated by

X L = AX H + ε (3.2.1)
where X L is the low-resolution image, X H is the high-resolution image, ε is a white Gaussian noise with the standard derivation σ, and A is an unknown super-resolution operator.

?figurename? 3.8: Sketch of the overall framework of SR.

In the following, we describe the estimation process of the forward operator A for super-resolution. For estimation, we assume that we have given a registered pair ( XH , XL ) of a high-resolution and a low-resolution image. We assume that our forward operator is given by

A = SH. (3.2.2)
Where H is a blur kernel and S is a downsampling operator. The downsampling operator S : R

m 1 ,n 1 → R m 2 ,n 2 (m 1 > m 2 and n 1 > n 2 ) is given by S = m 2 n 2 m 1 n 1 F -1 m 2 ,n 2 DF m 1 ,n 1 , (3.2.3)
where

D : R m 1 ,n 1 → R m 2 ,n 2 and for x ∈ R m 1 ,n 1 the (i, j)-th entry of D (x) is given by            x i,j , if i ≤ m 2 2 and j ≤ n 2 2 , x i+m 1 -m 2 ,j , if i > m 2 2 and j ≤ n 2 2 , x i,j+n 1 -n 2 , if i ≤ m 2 2 and j > n 2 2 , x i+m 1 -m 2 ,j+n 1 -n 2 , if i > m 2 2 and j > n 2 2 . F m 1 ,n 1 is the discrete two-dimensional Fourier transform (DFT) defined by F m 1 ,n 1 := F n 1 ⊗ F m 1 , where F m 1 = (exp (-2πikl/m 1 )) m 1 -1 k,l=0 .
The question now is how to estimate the blur kernel H. We assume that we have given a reference HR image XH ∈ R m 1 ,n 1 and a corresponding LR image XL ∈ R m 2 ,n 2 related by XL = A XH = S(H * XH )), where the blur kernel H is unknown. By the definition of downsampling operator S, we apply the DFT on both sides of XL = S(H * XH )), we obtain the following

F m 2 ,n 2 XL = m 2 n 2 m 1 n 1 D F m 1 ,n 1 H * XH = m 2 n 2 m 1 n 1 D (F m 1 ,n 1 (H))⊙D F m 1 ,n 1 XH , (3.2
.4) where ⊙ indicates the element-wise product. Now, we can conclude that

D (F m 1 ,n 1 (H)) = m 1 n 1 m 2 n 2 F m 2 ,n 2 XL ⊘ D F m 1 ,n 1 XH , (3.2.5)
where ⊘ is the element-wise quotient. In practice, we stabilize this quotient by increasing the absolute value of D F m 1 ,n 1 XH by 10 -5 while retaining the phase. Thus, assuming that the high-frequency part of H is negligible (i.e., that

D T DH = H), we can approximate F m 1 ,n 1 (H) by F m 1 ,n 1 (H) ≈ m 1 n 1 m 2 n 2 D -1 F m 2 ,n 2 XL ⊘ D F m 1 ,n 1 XH (3.2.6)
Therefore, the blur kernel H can be estimated by applying the inverse DFT

H ≈ m 1 n 1 m 2 n 2 F -1 m 1 ,n 1 D -1 F m 2 ,n 2 XL ⊘ D F m 1 ,n 1 XH (3.2.

7)

?chaptername? 3. Super-resolution for 3D images from materials science

where

F -1 m 1 ,n 1 is the inverse of F m 1 ,n 1 .

State-of-the-art of Super-resolution method

To date, mainstream algorithms of SISR are mainly divided into three families: the interpolation-based approach, the reconstruction-based approach, and the examplebased approach. We will briefly summarize their principles in the following.

Interpolation-based approach The most widely used interpolation methods in image processing and super-resolution are generally referred to as nearest neighbors, bilinear, bicubic [START_REF] Keys | Cubic convolution interpolation for digital image processing[END_REF], and Lanczos [START_REF] Duchon | Lanczos filtering in one and two dimensions[END_REF]. These methods compute the missing pixels in the HR image as averages of known pixels. However, some objects often show discontinuities, such as edges and corners. Thus, learning missing pixels leads to ringing, jagged edges, and blurred artifacts in the output image. Therefore, interpolation-based SISR methods are fast and straightforward, but suffer from accuracy deficiencies.

Reconstruction-based approach Reconstruction-based SR methods often use prior knowledge to improve the quality of reconstruction. This approach includes a variety of methods that focus on sharpening edges such as [START_REF] Dai | Soft Edge Smoothness Prior for Alpha Channel Super Resolution IEEE/CVF Conference on Computer Vision and Pattern Recognition[END_REF][START_REF] Dai | Softcuts: a soft edge smoothness prior for color image super-resolution[END_REF][START_REF]Image Upsampling via Imposed Edge Statistics[END_REF][START_REF] Aly | Image up-sampling using total-variation regularization with a new observation model[END_REF][START_REF] Marquina | Image super-resolution by TV regularization and Bregman iteration[END_REF]. These methods motivate to design an edge smoothness prior and suppress the ringing, jagged edge artifact of the interpolation-based methods. In [START_REF] Dai | Soft Edge Smoothness Prior for Alpha Channel Super Resolution IEEE/CVF Conference on Computer Vision and Pattern Recognition[END_REF][START_REF] Dai | Softcuts: a soft edge smoothness prior for color image super-resolution[END_REF], S. Dai et al.

proposed methods that based on soft edge smoothness prior to extract the edges and corner points of LR image. Then they perform the continuity of these edges and combine with the interpolation results to yield the final HR image. Similarly, R.

Fattal proposed a method in [START_REF]Image Upsampling via Imposed Edge Statistics[END_REF] in which missing information is learned from edge statistics. Beside that [START_REF] Aly | Image up-sampling using total-variation regularization with a new observation model[END_REF][START_REF] Marquina | Image super-resolution by TV regularization and Bregman iteration[END_REF] extend to new formulation to solve the SR problem through regularization term. They use the total variation (TV) regularizer that specifies the requirements of edge-directed filtering. These reconstruction-based SR methods provide improved results, yielding sharp edges without ringing or other artifacts. However when the magnification factor increases, the performance of these methods degrade rapidly and they are often time consuming.

Example-based approach This third approach is arguably the most successful because of its fast computation and outstanding performance. The example-based SR methods use machine learning techniques to analyze statistical relationships between the LR and corresponding HR image from an image database. These methods can be typically achieved by dividing images into small (overlapping) patches. In the upsampling procedure, the LR input image is divided into a set of patches. Then for each LR input patch, a HR patch is estimated by using the statistical parameters, that are learned from the examples contained in the dictionary.

State-of-the-art of Super-resolution method

The first study on this approach was introduced by Freeman et al. [START_REF] Freeman | Example-based superresolution[END_REF], based on Markov Random Field (MRF) framework. For the input LR patch, by searching the nearest neighbors in the training set, several paired HR candidate patches will be selected. These HR candidate patches subsequently are used in the MRF to reconstruct output HR patch. A disadvantage of this method is its high computational cost, due to the complex solution and the necessity of having large dictionaries, including a large variety of image patches.

To address the problem of dictionary size, many example-based SR algorithms consisting of different procedures have been proposed. The neighbor embedding method [START_REF] Chang | Super-resolution through neighbor embedding[END_REF] proposed by Chang et al. utilized the local geometric similarities of LR and HR to restore HR patches. The general idea of this method is that each observed LR patch can be described as a linear combination of its nearest neighbors in the dictionary of LR patches. In the dictionary, we can select the HR patches corresponding to the nearest neighbors in the LR patches. Then the output HR patch can be reconstructed by a linear combination of the HR neighbors with the same linear weights obtained in LR. In this way, since patch combinations are enabled, the number of required image patch exemplars can ideally be reduced, while maintaining the same size of the dictionary. However, the requirement to determine the nearest neighbors for each patch from a large example dictionary might render the computation too expensive if the dictionary size is large and the zoom factor is high.

Another important line of example-based SR methods was initiated in sparsecoding SR (ScSR) method [START_REF] Yang | Image Super-Resolution Via Sparse Representation[END_REF] using sparse representation of image patches over the joint dictionaries. These joint dictionaries {D h , D l } include pairs of HR and LR patches such that the sparse representation α of the HR patch is the same as the sparse representation of the corresponding LR patch. Finally, for a given input LR patch y i , its sparse representation α i is first computed using D l and the corresponding HR patch is estimated as xi = D h α i . Due to joint learning, the estimated HR and LR dictionaries in ScSR are not well adapted to their respective feature spaces. Some SR algorithms were proposed in [START_REF] Zeyde | On single image scale-up using sparserepresentations[END_REF][START_REF] Wang | Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis[END_REF][START_REF] Yang | Coupled dictionary training for image super-resolution[END_REF] by modifying ScSR, using coupled dictionaries. These sparse coding methods are an improvement over neighbor embedding and provide impressive results. Although the dictionary size is significantly reduced, the execution time of the ScSR method is still long.

In recent years, many deep learning (DL)-based SISR methods have been proposed in the literature. In fact, the use of neural networks has demonstrated their effectiveness in producing state-of-the-art results. Dong et al. proposed a SR algorithm named SRCNN in [START_REF] Dong | Learning a deep convolutional network for image super-resolution[END_REF][START_REF] Dong | Image super-resolution using deep convolutional networks[END_REF] that uses a three-layered convolutional neural network (CNN) to learn end-to-end mapping between high and low-resolution images. The input of this network is a HR image interpolated from the LR image based on Bicubic interpolation method. This leads to erroneous estimates of the image structure when the inputs have been smoothed by interpolation. To overcome this issue, some research uses the deconvolution network, which was introduced by Zeiler et al. [START_REF] Zeiler | Adaptive deconvolutional networks for mid and high level feature learning[END_REF], to learn an upsampling operator. Fast SRCNN (FSRCNN) method proposed in [START_REF] Dong | Accelerating the super-resolution convolutional neural network[END_REF] by Dong et al. is the pioneering work using the deconvolution ?chaptername? 3. Super-resolution for 3D images from materials science layer to reconstruct HR images directly from LR images. Using the deconvolution layer at the end of the network, the mapping is learned directly from the original low-resolution image (without interpolation) to the high-resolution one. This leads to a significant reduction in computation of the SRCNN proposal. However, the deconvolution layer enlarges feature maps by padding the images with zeros and then does the convolution, which may not be good for the results. Besides that, Shi et al. proposed an efficient sub-pixel convolution layer in [START_REF] Shi | Real-time single image and video superresolution using an efficient sub-pixel convolutional neural network[END_REF], known as efficient sub-pixel convolutional neural network (ESPCN). ESPCN expands the multiple-channel feature maps to store the extra points to increase resolution and then rearranges these points to obtain the HR output. With all these adversarial networks, SRGAN significantly improves the overall visual quality of reconstruction over previous methods. Along with the promising performance that deep learning algorithms have achieved in SISR, there remain several important challenges and drawbacks such as the expensive computation of the depth architecture, the vagueness of the optimization objectives, the unknown corruption and large upsampling factor of SR.

Conclusion

This chapter provides a general overview of super-resolution processing as well as issues of super-resolution for 3D materials data. Firstly, this chapter introduces our material data and gives some challenges when performing a super-resolution for 3D materials data. The three main challenges are detecting the locality of the HR image in the corresponding LR image, changing the contrast between the HR and LR images, and modeling of an unknown downsampling operator. Second, we introduce and classify the methods that can be used to perform super-resolution. Three main groups have been presented as interpolation-based approaches, reconstruction-based approaches, and example-based approaches. Interpolation methods allow one to resample a low-resolution image at different sampling rates. However, the highresolution image intensity values are a fixed linear function of the low-resolution image. Therefore, only low spatial frequencies are reconstructed in the reconstructed image. This leads to many artifacts in the output image. Reconstruction-based SR methods provide results without ringing or other artifacts and improve interpolation methods significantly. However, the execution time of these method is too large when the size of image and the magnification factor increases. To improve the results and execution time, example-based methods were first introduced by Freeman et al. [START_REF] Freeman | Example-based superresolution[END_REF] using some machine learning techniques This chapter aims to present the modeling of the super-resolution problem guided by high-resolution local sub-images from an example-based point of view.

First, a brief overview of the example-based approaches used is provided. A SR method based on sparse representation of image patches over the joint dictionaries and particularly of a joint Gaussian mixture model (GMM) is followed by a stateof-the-art in order to describe the framework of this thesis.

Then an extension method is proposed using the MMSE with the GGMM to reconstruct HR image.

This chapter is organized as follows. A short summary of patch-based SR approaches is provided in Section 4.1. The MMSE estimator with GGMM [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] and EPLL method [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF] are mentioned in this section. Section 4.2 proposes a method that uses the MMSE estimator for the joint GGMM, called the MMSE-GGMM method. This is a main contribution in this chapter, and it adapts the FP-EM algorithm to learn the joint GGMM. Finally, the conclusion is given in Sections 4.3.

?chaptername? 4. GMM and GGMM for Super-resolution

State of the art of patch-based approach

Super-resolution is the task of reconstructing the estimate XHR of a HR image X H based on a LR observation X L . The observed LR image is generated by

X L = AX H + ε (4.1.1)
where ε is a white Gaussian noise with the standard derivation σ, and A is an unknown downsampling operator.

In recent years, various patch-based super-resolution image algorithms have been presented, and their efficiency with respect to image super resolution has been studied. Zoran and Weiss [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF] proposed using the negative log-likelihood function of a GMM as a regularizer of the inverse problem. The estimated HR image XHR is computed by solving arg min

X H ∥AX H -X L ∥ 2 -λ i∈I log p (X H,i ) (4.1.2)
where p is the probability density function of the GMM and (X H,i ) i∈I are the patches in the HR image. This method is called EPLL. However, EPLL requires knowledge of the operator A, which is not the case in some real applications. Therefore, we investigate the alternative approach of P. Sandeep et al. [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF], which uses a joint GMM of the concatenated vectors of HR and corresponding LR patches. Each HR patch is estimated from the LR patch by using the Minimum Mean Squared Error MMSE estimator as:

XH ∈ arg min T E∥X HR -T (X L ) ∥ 2 2 (4.1.3) = µ H + Σ HL Σ -1 L (X L -µ L ) . (4.1.4)
In the following, we detail the SR method using EPLL method and joint GMM with MMSE.

Super-resolution via Joint Mixture Models based on MMSE [75]

In this section, we briefly revisit the super-resolution by Sandeep wt al. [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] using joint Gaussian mixture models based on MMSE. Let {P θ : θ ∈ Θ} be a parametric family of probability distributions with density functions p(•|θ). Then, a mixture model is a probability distribution defined by the probability density function

F (x|w, Θ) = K k=1 w k p(x|θ k ), w = (w 1 , ..., w K ), Θ = (θ 1 , ..., θ K ), (4.1.5)
where the weights w k are non-negative and sum up to one. If θ = (µ, Σ) and P θ = N (µ, Σ), we speak of a Gaussian mixture model (GMM). The (absolutely continuous) d-dimensional normal distribution N (µ, Σ) with mean µ ∈ R n and positive semi-definite covariance matrix Σ ∈ SPD(n) has the density

f (x|µ, Σ) = (2π) -n 2 abs Σ -1 2 exp - 1 2 (x -µ) T Σ -1 (x -µ) . (4.1.6)
Throughout this work, we aim to reconstruct the unknown high-resolution image X H based on a low resolution observation X L . Here, we assume that we have given a reference image, where the high-resolution image xH as well as the low-resolution observation xL are both known.

In this setting, Sandeep and Jacob proposed in [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] to reconstruct X H using the following steps.

1. Learning a joint mixture model. In a first step, for given low-resolution patches xL,i ∈ R τ 2 of an image and their high-resolution counterparts xH,i ∈ R q 2 τ 2 , q ∈ N, q ≥ 2, i = 1, ..., N , we approximate the distribution of the joint high-and low-resolution patches

x i = xH,i xL,i ∈ R n , n = (q 2 + 1)
w = (w k ) k , µ = (µ k ) k , Σ = (Σ k ) k with µ k = µ H,k µ L,k , Σ = Σ H,k Σ HL,k Σ T HL,k Σ L,k .
2. Estimation of the high-resolution patches using the MMSE estimator.

In the second step, we estimate the high-resolution correspondence of a given low-resolution patch x L ∈ R τ 2 . For this purpose, we first select the component k * , such that the likelihood that x L belongs to the k * -th component is maximal, i.e., we compute

k * = arg max k=1,...,K w k p(x L |µ L,k , Σ L,k )
Afterwards, we estimate the high-resolution patch x H ∈ R q 2 τ 2 as the minimum mean square estimator (MMSE) of X H given X L = x L for a random variable X = (X H , X L ) ∼ P θ k . For Gaussian distributions, this corresponds to

xH = µ H,k + Σ HL,k Σ -1 L (x L -µ L,k ).
We give a more detailed explanation on MMSE estimators in Section 4.2.2.

Reconstruction of the high-resolution image by patch averaging.

Finally, we estimate for any patch within the low-resolution image, the highresolution correspondence. Afterwards, we reconstruct the high-resolution ?chaptername? 4. GMM and GGMM for Super-resolution image as follows: Let x H = (x k,l ) qτ k,l=1 ∈ R qτ,qτ be a two-dimensional highresolution patch. Then, we assign to each pixel x k,l the weight

ρ k,l := exp -γ 2 (k -qτ +1 2 ) 2 + (l -qτ +1 2 ) 2 .
After that, we add up for each pixel in the high-resolution image the corresponding weighted pixel values and normalize the result by dividing by the sum of the weights.

For readability, due to the combination of MMSE estimator with GMM, in the following of this thesis, we call this method is MMSE-GMM. This method outperforms the ScSR [START_REF] Yang | Image Super-Resolution Via Sparse Representation[END_REF] algorithm with a significant margin. Instead of learning the pair of dictionaries from the reference HR-LR patch pairs as ScSR algorithm, the MMSE-GMM method learns the joint GMM from the reference images. This can improve the speed and performance of the MMSE-GMM algorithm.

Expected patch log-Likelihood (EPLL) method for superresolution [92]

The second approach of patch-based SR approaches, which we would like to mention in this thesis, is the Expected patch log Likelihood (EPLL) method by Zoran and Weiss [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF]. The EPLL of a given image X H is defined under prior p as:

EP LL p (X H ) = i∈I log p (P i X H ) , (4.1.7)
where I = {1, ...N } is a set of pixel indices; P i : R N → R P is a linear operator which extracts the i-th patch from the image X H , and log p (P i X H ) is the likelihood of the i-th patch under the prior p. The basic idea of this method is to maximize the EPLL while the AX H is still being close to the low-resolution image X L . Thus the EPLL method defines the restored image as the maximum a posteriori estimate, corresponding to the following minimization problem:

arg min X H λ 2 ∥AX H -X L ∥ 2 - i∈I log p (P i X H ) . (4.1.8)
This optimization is a non-convex problem and depends on prior of the operator P i (x). Thus the problem therefore becomes difficult to calculate. In order to avoid the optimization of the prior, Zoran et al. use a classical method, known as "Half Quadratic Splitting" [START_REF] Geman | Nonlinear image recovery with half-quadratic regularization[END_REF][START_REF] Krishnan | Fast image deconvolution using hyper-laplacian priors[END_REF]. This technique introduces a set of patches {z i } N i=1 ⊂ R P , one for each overlapping patch P i X H in the image, and consider instead the penalized optimization problem that reads, for β > 0, as:

arg min X H ,z 1 ,...z N λ 2 ∥AX H -X L ∥ 2 + i∈I β 2 ∥P i X H -z i ∥ 2 -log p (z i ) . (4.1.9)
Equation (4.1.9) for a fixed β value is solved by an alternating optimization scheme:

• Solving X H while keeping {z i } i∈I constant:

XH ∈ arg min X H λ 2 ∥AX H -X L ∥ 2 + β 2 i∈I ∥P i X H -z i ∥ 2 (4.1.10) • Solving z i for all i = 1, ..., N given X H : ẑi ∈ arg min z i β 2 ∥P i X H -z i ∥ 2 -log p (z i ) (4.1.11)
Sub-problem (4.1.10) corresponds to solving a linear inverse problem with a Tikhonov regularization, and has an explicit solution often referred to as Wiener filtering:

XH = λA T A + β i∈I P T i P i -1 λAX L + β i∈I P T i z i . (4.1.12)
The solution to Sub-problem (4.1.11) depends on the prior p and the sub-problem is considered as a Maximum A-Posteriori estimator (MAP). In the case of Gaussian mixture model, [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF] keeps only 1 component k * of Gaussian mixture model maximizing the likelihood for the given i-th patch. With this assumption, the solution of (4.1.11) is also given by Wiener filtering:

z i = Σ k * z i + 1 β Id P -1 Σ k * z i zi + 1 β Id P µ k * z i . (4.1.13)
In the same setting reference images and testing images of MMSE-GMM method, the EPLL method can reconstruct X H with GMM as follows:

1. Learning Gaussian mixture model. In a first step, for given highresolution patches xH,i ∈ R q 2 τ 2 , q ∈ N, q ≥ 2, i = 1, ..., N , we approximate the distribution of the joint high-resolution patches by Gaussian mixture model such that the resulting approximation is characterized by the parameters

w = (w k ) k , µ H = (µ H,k ) k , Σ = (Σ H,k ) k .
2. Estimation of the patches {z i } i∈I using the MAP estimator. In the second step, we first select the component k * , such that the likelihood that x L belongs to the k * -th component is maximal, i.e., we compute

k * = arg max k=1,...,K w k p(x L |µ L,k , Σ L,k ).
?chaptername? 4. GMM and GGMM for Super-resolution

We estimate z i for all i ∈ I while give the HR image X H by using Equation (4.1.13).

3. Reconstruction of the high-resolution image. Finally, we estimate for the high-resolution image X H by using Equation (4.1.12).

Conclusion

In both MMSE-GMM and EPLL-GMM methods, the GMM is used to estimate the high-resolution patches. While the MMSE learns the joint GMM of the concatenated vector of high-and low-resolution patches, the EPLL only needs to learn the GMM of the high-resolution patches. By using the knowledge of A, the EPLL can achieve better results than MMSE. However, in some cases, such as our material data, we have no knowledge of operator A. Thus we have to learn the operator A from the reference high-and low-resolution images, as discussed in Subsection 3.2.3. This means that the MMSE method can reconstruct HR image without the knowledge of operator A. Therefore, we decide to learn the SR method by extending the MMSE estimator with the joint mixture model. In the following section, we will propose a method that is based on MMSE estimator with joint generalized Gaussian mixture model. In this section, we will demonstrate that our method does not need the knowledge of operator A and is invariant with the contrast change problem.

Contribution: MMSE estimator with generalized

Gaussian mixture model for super-resolution [START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF] In the second step of MMSE-GMM method, the high-resolution patches is estimated thanks to the parameters of the joint GMM model. GMM can be learned by the expectation-maximization algorithm (EM) [START_REF] Byrne | The EM Algorithm: Theory, Applications and Related Methods[END_REF][START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF]. In the past decade various works [START_REF] Allili | Wavelet modeling using finite mixtures of generalized Gaussian distributions: Application to texture discrimination and retrieval[END_REF] [10] [START_REF] Pascal | Parameter estimation for multivariate generalized Gaussian distributions[END_REF] have shown that generalized Gaussian mixture model (GGMM) has more flexibility to adapt to the shape of data probability density function and less sensibility for over fitting the number of classes than the GMM. In [START_REF] Deledalle | Image denoising with generalized Gaussian mixture model patch priors[END_REF], C. Deledalle et al. proposed a method that uses the generalized Gaussian mixture model (GGMM) for the EPLL algorithm (EPLL-GGMM). They showed that the GGMM gets the distribution of patches better than a GMM and that it outperforms the GMM when used in the EPLL framework. However, the EPLL-GGMM model learns the parameters of the mixture model by estimating the covariance matrix with the following formula:

Σ k = N i=1 ξ i,k xH,i xT
H,i N i=1 ξ i,k . (4.2.1)
Here {x H,i } N i=1 is a set of all training HR patches and ξ i,k is the conditional distribution of the component k given the patch xH,i . Unfortunately, this formula is not true in the GGMM setting.

Contribution: MMSE estimator with generalized

Gaussian mixture model for super-resolution [START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF] 35

In [START_REF] Boukouvalas | A new Riemannian averaged fixed-point algorithm for MGGD parameter estimation[END_REF], Z. Boukouvalas et al. introduced a Riemannian averaged Fixed-point (RA-FP) algorithm for multivariate generalized Gaussian distribution (MGGD) parameter estimation. The RA-FP algorithm can effectively estimate the scatter matrix for any value of the shape parameter: for instance, β = 1 is the Gaussian distribution and β = 0.5 is the Laplacian distribution.

In the case of the generalized Gaussian mixture model, we generalize the algorithm from [START_REF] Wang | Globally Convergent Algorithms for Learning Multivariate Generalized Gaussian Distributions[END_REF] for the weighted maximum likelihood estimation using the EM algorithm based on a fixed-point algorithm.

This algorithm estimates the parameters of the mixture model, including the shape parameters of each component. This algorithm is called FP-EM algorithm for generalized Gaussian mixture model. In our work, we provide a method that uses the MMSE estimator for the super-resolution based on GGMM for joint HR-LR modeling, called MMSE-GGMM.

Parameter Estimation of Mixture Models

In this subsection, we focus on the parameter estimation of mixture models. For this purpose, we aim to employ the expectation-maximization EM algorithm. First, in Subsection 4.2.1.1, we revisit the generic form of the EM algorithm for mixture models. This algorithm contains as a subproblem the weighted maximum likelihood estimation for the corresponding parametric distribution family. Thus, we consider this subproblem in Subsection 4.2.1.2 for some elliptical distributions, namely, the Gaussian distribution, the generalized Gaussian distribution.

EM algorithm for Mixture Models

In the following, we consider the EM algorithm to estimate the parameters of the mixture models. EM algorithms were introduced in [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] and can be used for various probability distributions. We refer to [START_REF] Byrne | The EM Algorithm: Theory, Applications and Related Methods[END_REF] for an overview. Given samples x 1 , ..., x N aims to minimize the negative log-likelihood function

L(w, Θ) = - 1 N N i=1 log(F (x i |w, Θ),
where F is defined as in (4.1.5). Then, the EM algorithm for mixture models reads as Algorithm 1, see e.g. [START_REF] Mclachlan | Finite Mixture Models[END_REF].

As outlined in the algorithm, we have to compute the weighted maximum likelihood estimator as a sub-problem of the EM algorithm, i.e., we need to maximize

ℓ(θ) = N i=1 α i log(p(x i |θ)) (4.2.5)
for the underlying parametric distribution family. For readability, in the following, we denote α i,k by α i and θ k by θ. In the following sections, we show how the EM algorithm can be done for various distributions.

Algorithm 1 EM Algorithm for Mixture Models

Input: x = (x 1 , ..., x N ) ∈ R n×N , initial estimate w (0) , Θ (0) . for r = 0, 1, ... do E-Step: For k = 1, ..., K and i = 1, . . . , N compute In order to solve subproblem (4.2.5) within the EM algorithm for mixture models, we consider in the following the weighted maximum likelihood estimation for several elliptical distributions. Let SPD(n) be the set of positive-definite symmetric matrices. A probability distribution E n (µ, Σ, g) with µ ∈ R n and Σ ∈ SPD(n) is called an elliptical distribution if it admits a density function

α (r) i,k = w (r) k p(x i |θ (r) k ) K j=1 w (r) j p(x i |θ (r) j ) (4.2.2) M-Step: For k = 1, ..., K compute w (r+1) k = 1 N N i=1 α (r) i,k , (4.2.3)
θ (r+1) k = arg max θ k N i=1 α (r) i,k log(p(x i |θ k )) . ( 4
p(x|µ, Σ, g) ∝ Σ -1/2 g((x -µ) T Σ -1 (x -µ)),
where g : R ≥0 → R ≥0 is Lebesgue measurable with

∞ 0 t n/2-1 g(t)dt < ∞.
In this work, we consider Gaussian, generalized Gaussian as examples of elliptical distributions.

Gaussian distribution. A probability distribution is a Gaussian distribution N (µ, Σ) with mean µ ∈ R n and covariance matrix Σ ∈ SPD(n) if it has the probability density function

p(x|µ, Σ) = 1 (2π) n/2 |Σ| 1/2 exp -1 2 (x -µ) T Σ -1 (x -µ) .
The Gaussian distribution is an elliptical distribution with g(t) = exp (-1 2 t). For samples x 1 , ..., x N with weights α 1 , ..., α N , we can calculate the weighted maximum 4.2. Contribution: MMSE estimator with generalized Gaussian mixture model for super-resolution [START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF] 37 likelihood estimator of µ and Σ, i.e., the solution of (4.2.5) by setting the gradients of the objective function to zero. This leads to estimators

μ = 1 N i=1 α i N i=1 α i x i (4.2.6) Σ = 1 N i=1 α i N i=1 α i (x i -μ)(x i -μ) T = 1 N i=1 α i N i=1 α i x i x T i -μμ T . (4.2.7)
Generalized Gaussian distribution. We call a probability distribution a generalized Gaussian distribution GG(µ, Σ, β) with expected value µ ∈ R n , scatter matrix Σ ∈ SPD(n) and shape parameter β ∈ R >0 , if it has the probability density function

p(x|µ, Σ, β) = C(β) |Σ| 1/2 exp -1 2 (x -µ) T Σ -1 (x -µ) β , (4.2.8)
where the normalizing constant C is given by

C(β) = βΓ( n 2 ) 2 n/(2β) π n/2 Γ( n 2β )
.

Here, Γ denotes the gamma function. For small values of the shape parameter β, we obtain heavy-tailed distributions, while large values of β mean that the corresponding generalized Gaussian distribution is flat-tailed. The generalized Gaussian distribution includes the Gaussian distribution for β = 1 and the Laplace distribution for β = 1/2. The generalized Gaussian distribution is an elliptical distribution with g(t) = exp (-1 2 t β ), β > 0. In the literature, several algorithms have been proposed to estimate the parameters of a generalized Gaussian distribution of samples x 1 , ..., x N . Most of these methods are based on fixed point (FP) iterations [START_REF] Pascal | Parameter estimation for multivariate generalized Gaussian distributions[END_REF][START_REF] Najar | A Fixed-point Estimation Algorithm for Learning The Multivariate GGMM: Application to Human Action Recognition[END_REF] or first-order computations on the Riemannian manifold of parameters [START_REF] Boukouvalas | A new Riemannian averaged fixed-point algorithm for MGGD parameter estimation[END_REF]. For this work, we generalize the algorithm from [START_REF] Wang | Globally Convergent Algorithms for Learning Multivariate Generalized Gaussian Distributions[END_REF] for weighted maximum likelihood estimation (4.2.5), i.e., for maximizing

ℓ(µ, Σ, β) = N i=1 α i log(p(x i |µ, Σ, β)),
for some weights α 1 , ..., α N > 0. The following proposition gives the computations for estimating parameters of generalized Gaussian distribution.

Proposition 1. Let GG(µ, Σ, β) be a generalized Gaussian distribution with a density function q(•|µ, Σ, β). Any maximizer (µ, Σ, β) of the function ℓ(µ, Σ, β) ?chaptername? 4. GMM and GGMM for Super-resolution should satisfy the following equations:

µ = N i=1 α i δ β-1 i x i N i=1 α i δ β-1 i , (4.2.9)
Σ = N i=1 α i δ β-1 i (x i -µ)(x i -µ) T N i=1 α i , (4.2.10
)

β = β + ρ∇ β ℓ(µ, Σ, β), (4.2.11)
where δ i = (x i -µ)Σ -1 (x i -µ) and ρ > 0.

?proofname? In the following, we compute the derivatives of

ℓ(µ, Σ, β) := N i=1 α i log f β (x i |µ, Σ), (4.2.12)
which is up to a constant equal to

N i=1 α i log C p (β) |Σ| 1/2 exp - 1 2 (x i -µ) T Σ -1 (x i -µ) β (4.2.13) (4.2.14)
Using the notation δ i = (x i -µ) T Σ -1 (x i -µ), the gradient with respect to µ is given by

∇ µ ℓ(µ, Σ, β) = N i=1 α i βΣ -1 (x i -µ)δ β-1 i (4.2.15)
Setting the gradient to zero yields the following result.

µ = N i=1 α i x i δ β-1 i N i=1 α i δ β-1 i . (4.2.16)
Plugging in the formulas

∂a T Σ -1 b ∂Σ = Σ -T ab T Σ -T and ∂ log |Σ -1 | ∂Σ = ∂ log |Σ -1 | ∂Σ -1 ∂Σ -1 ∂Σ = ΣΣ -2 = Σ -1
from [START_REF] Petersen | The Matrix Cookbook[END_REF], we obtain that

∇ Σ ℓ(µ, Σ, β) = N i=1 α i 1 2 Σ -1 - β 2 δ β-1 i Σ -1 (x i -µ)(x i -µ) T Σ -1 .
Setting the gradient to zero and multiplying by Σ from the left and from the right 4.2. Contribution: MMSE estimator with generalized Gaussian mixture model for super-resolution [START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF] 39

yields that Σ = N i=1 α i δ β-1 i (x i -µ)(x i -µ) T N i=1 α i .
Finally, the gradient with respect to β is given by

∇ β ℓ(µ, Σ, β) = N i=1 α i ∇ β log(C n (β)) -1 2 ∇ β δ β i
Here, we have that

∇ β δ β i = δ β i log(δ i ) and ∇ β log(C n (β)) = 1 β + n 2β 2 ψ( n 2β ) + log 2
, where ψ is the digamma function. Hence, we obtain

∇ β ℓ(µ, Σ, β) = 1 β + n 2β 2 ψ( n 2β ) + log 2 N i=1 α i + N i=1 α i δ β i log(δ i ).
The expression of maximizer (µ, Σ, β) motivate us to maximize ℓ (µ, Σ, β) by generating a sequence (µ (r) , Σ (r) , β (r) ) by fixed-point (FP) iteration.

µ (r+1) = N i=1 α i (δ (r) i ) β (r) -1 x i N i=1 α i (δ (r) i ) β (r) -1 , (4.2.17)
Σ (r+1) = N i=1 α i (δ (r) i ) β (r) -1 (x i -µ (r) )(x i -µ (r) ) T N i=1 α i , (4.2.18
)

β (r+1) = β (r) + ρ∇ β ℓ(µ (r) , Σ (r) , β (r) ), (4.2.19)
where δ

(r) i = (x i -µ (r) )(Σ (r) ) -1 (x i -µ (r)
). Note that ∇ β ℓ(µ, Σ, β) can be explicitly derived as

∇ β ℓ(µ, Σ, β) = 1 β + n 2β 2 ψ( n 2β ) + log 2 N i=1 α i + N i=1 α i δ β i log(δ i ).
The EM algorithm with the FP iteration for GGMM is called FP-EM algorithm.

Remark 1. In the unweighted case, i.e., α 1 = . . . = α N , the authors of [START_REF] Wang | Globally Convergent Algorithms for Learning Multivariate Generalized Gaussian Distributions[END_REF] show convergence of the above iteration.

Remark 2. Note that for the estimation of β, several other approaches could be considered such as the Newton-Raphson method [START_REF] Ypma | Historical development of the Newton-Raphson method[END_REF].

MMSE Estimator for Generalized Gaussian Distributions

In the following, we consider the computation of a high-resolution patch x H given the corresponding low-resolution patch x L and a mixture model (4.1.5), where the distribution P θ k is a generalized Gaussian distribution. In particular, we have that the parameters are given by θ k = (µ k , Σ k , β k ) for some location µ k ∈ R n , a scatter matrix Σ k ∈ SPD(n) and β > 0.

As in [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF], we first split the parameters

µ k = µ H,k µ L,k , Σ k = Σ H,k Σ HL,k Σ T HL,k Σ L,k
into components belonging to the high-and low-resolution part of the distribution p(•|θ k ). Then we select the component k * , which best matches the low-resolution patch x L by

k * := arg max k=1,...,K p(x L |µ L,k , Σ L,k , β k ).
Subsequently, we compute x H as the minimum mean square estimator (MMSE) of X H given X L = x L , where X = (X H , X L ) ∼ P θ k .

Remark 3. Given a random variable X L : Ω → R d in a probability space (Ω, A, P), we wish to estimate a random variable X H : Ω → R D , i.e., we seek an estimator T : R d → R D such that XH = T (X L ) approximates X H . A common quality measure for this task is the mean square error E∥X H -T (X L )∥ 2 2 , which gives rise to the definition of the minimum mean square estimator

T MMSE ∈ arg min T E∥X H -T (X L )∥ 2 2 . (4.2.20)
Under weak additional regularity assumptions on the estimator T , the Lehmann-Scheffé theorem [START_REF] Lehmann | Completeness, similar regions, and unbiased estimation: Part I[END_REF] states that the general solution of the minimization problem (4.2.20) is given by

T MMSE (X L ) = E(X H |X L ).
In general, it is not possible to give an analytical expression of the MMSE estimator T MMSE . In the case of elliptical distributions, the following theorem can be found, e.g., in [START_REF] Gomez | A survey on continuous Elliptical vector distributions[END_REF]Theorem 8].

Theorem 1. Assume that X = (X H , X L ) : Ω → R n has an elliptical distribution P θ with parameters θ = (µ, Σ, g), where

µ = µ H µ L , Σ = Σ H Σ HL Σ T HL Σ L
Then, for each P X L -almost every x L , we have that the conditional distribution P X H |X L =x L is given by the elliptical distribution P θ, where the parameters θ = (μ, Σ, ĝ) are given by

μ = µ H + Σ HL Σ -1 L (x L -µ L ), Σ = Σ H -Σ HL Σ -1 L Σ T HL , ĝ(t) = g(t + t 0 ) with t 0 = (x L -µ L ) T Σ -1 L (x L -µ L ).
Since a generalized Gaussian distribution is also an elliptical distribution, the

T MMSE (X L ) = E(X H |X L ) = µ H + Σ HL Σ -1 L (x L -µ L ). (4.2.21)
In our super-resolution task, we assume that the vector

x H x L is a realization of the random variable X H X L ∼ N (µ k * , Σ k * ).
Then, by (4.2.21), the MMSE can be computed as

x H = µ H,k * + Σ HL,k * Σ -1 L,k * (x L,k * -µ L,k * ). (4.2.22)
Finally, we can apply the reconstruction of the high-resolution image step by patch averaging as in 3, Subection 4.1.1.

In some applications, the high-resolution image and its low-resolution correspondence have different brightness and contrast. The following theorem shows that the MMSE estimator is invariant under affine transformations of the observation. In particular, the MMSE estimator compensates for the change in contrast and brightness.

Theorem 2. Assume that X = (X H , X L ) : Ω → R n has a generalized Gaussian distribution GG(µ, Σ, β), where µ = µ H µ L , Σ = Σ H Σ HL Σ T HL Σ L
Further define the random variable XL := α 1 X L + α 2 with α 1 > 0, α 2 ∈ R. Then it holds

E(X H | XL ) = E(X H |X L ) = µ H + Σ HL Σ -1 L (X L -µ L ). ( 4

.2.23)

?proofname? The first equality holds true as it holds

σ( XL ) = σ({ X-1 L (] -∞, x]) : x ∈ R d }) = σ({ X-1 L (] -∞, α 1 x + α 2 ]) : x ∈ R d }) (4.2.24) = σ({X -1 L (] -∞, x]) : x ∈ R d }) = σ(X L ). (4.2.25)
The second equality follows directly from Theorem 1.

Numerical Results

In this section, we demonstrate the performance on some 2D images such as Gold-hill, Barbara, Camera-man, and our material data, which was presented in Chapter. 3. All the implementations were done in Matlab. We run all experiments on PlaFRIM1 with 32 cores (4 Go/core) AMD EPYC 7452.

?chaptername? 4. GMM and GGMM for Super-resolution

× τ : {x L,i } N i=1 with xL,i ∈ R τ 2 .
For the HR image X H , we can also have a set of qτ × qτ HR patches {x H,i } N i=1 where xH,i ∈ R q 2 τ 2 corresponds to the LR patch xL,i . Finally, we define a set {x i } with a vector x i ∈ R τ 2 (q 2 +1) by concatenating the HR and LR patches as where M SE(X, Y ) is the mean square error between X and Y with d the peak value, corresponding to the maximum fluctuation in the image.

x i = xH,i xL,i ∈ R n . ( 4

MMSE-GGMM

First, to underline the advantages of the proposed method MMSE-GGMM, we compare our method with the MMSE-GMM [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] and the EPLL-GMM [START_REF] Zoran | From learning models of natural image patches to whole image restoration[END_REF] as well as EPLL-GGMM [START_REF] Deledalle | Image denoising with generalized Gaussian mixture model patch priors[END_REF]. In this subsection, we use all models in standard images of size 512 × 512, such as Gold Hill, Camera-Man, Barbara, and material images, for the magnification factor q = 2. Low-resolution images are created using the operator A in (3.2.2). To estimate the parameters of the mixture model, we extract the low-resolution image with the size of the overlap patches τ = 4. Thus, the corresponding high-resolution patch has size qτ = 8. Then we get N ≈ 1500 patches for the training data to learn the mixture model. Finally, high-resolution images can be reconstructed from the learned mixture model and LR images based on the MMSE as described in Subsection 4.1.1. To implement the EPLL method, we estimate the mixture models using the EM algorithm, which is introduced in [START_REF] Deledalle | Image denoising with generalized Gaussian mixture model patch priors[END_REF].

Once again, EPLL requires the knowledge of the operator A in the image estimation step. For the standard image experiments, we reconstruct the HR image with the operator A defined by equation (3.2.2), called ("EPLL with given A"). Besides that, we implement the EPLL method with the operator A estimated as in [START_REF] Altekruger | WPPNets: Unsupervised CNN Training with Wasserstein Patch Priors for Image Superresolution[END_REF] from the reference HR and LR images, called ("EPLL with learned A"). Since the operator A is learned from the quarter-known image, it is the same size as the reference image. Thus, we upsample the operator A by a zero padding technique to have the same size as the whole image. Table 4.1 shows the performance of the MMSE and EPLL methods in synthetic images. Both of these methods are compared based on the GMM, LMM, GGMM with fixed shape parameter β (e.g. β = 0.8, 0.4, 0.25) and the GGMM model which learns the shape parameter based on FP-EM algorithm. The PSNR values of MMSE-GGMM with the FP algorithm are slightly higher than those of the MMSE-GMM and LMM models. To compare our EM algorithm for the GGMM model with the EM algorithm proposed by Deledalle et al. [START_REF] Deledalle | Image denoising with generalized Gaussian mixture model patch priors[END_REF], we compare the MMSE method with the FP algorithm (MMSE-GGMM FP) and Deledalle's approach (MMSE-GGMM Deledalle) in Table 4.1. The MMSE-GGMM FP gives better results than the MMSE-GGMM Deledalle. Furthermore, we observed that our method can obtain results similar to those of EPLL with given A, while we do not have to learn the superresolution operator A. Besides the results of our method are 1.5 dB higher than EPLL with the learned operator A.

?chaptername? 4. GMM and GGMM for Super-resolution ?tablename? 4.1: PSNRs of the reconstructions of synthetically downsampled 2D images using either MMSE and EPLL approaches for GMM and LMM, GGMM with different shape parameter β. The magnification factor is q = 2.

Hill

In practice, LR and HR images were acquired under different conditions. Therefore, they may have different contrasts. Thus, we have to estimate the contrast change parameters between the LR and HR images. However, Theorem 2 proves that MMSE-GGMM is invariant for an affine contrast change. Therefore, MMSE-GGMM does not require learning these parameters and knowledge of the operator A, while the EPLL method does. To demonstrate these arguments, we perform the MMSE and the EPLL approach for our material data. Table 4.2 gives the PSNR value of the HR reconstructions of real material images using the MMSE and EPLL approach. The second column group gives the PSNR values of the EPLL approach with a learned operator A, which is estimated from the known part of the HR image. The PSNR values in the third column group are obtained from the EPLL method with the learning contrast parameters and the operator A. This operator is learned as in [START_REF] Altekruger | WPPNets: Unsupervised CNN Training with Wasserstein Patch Priors for Image Superresolution[END_REF] from the observed LR image and the whole HR image of the ground truth. We call this approach is "EPLL with given A". The PSNRs in Table 4.2 show that our method achieves significantly better results than the EPLL method with the learning A from the known HR part. Our method gives PSNRs 1 dB lower than the EPLL method with learned A, while knowledge of the operator A is not required in our method.

Conclusion

This chapter proposed a new algorithm to perform image super-resolution. We extended the image super-resolution using the GMM method provided by Sandeep and Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] to the GGMM model, which is learned by the FP algorithm. We also derived a new method based on a Fixed Point approach to estimate the parameters of GGMM model, which is called FP-EM algorithm. Experiments on synthetic and material images demonstrate that our method is a promising solution for image SR.

However, the EM algorithm for these models (GMM or GGMM) becomes very slow as the number of data points becomes large and the 3D images lead to an increase in the dimensionality of the training data. Therefore, one perspective to improve this method would be to use a dimensionality reduction method.

With this in mind, the next chapter takes into account that we incorporate a dimensionality reduction within mixture models of generalized Gaussian distributions and derive an algorithm for estimating its parameters.

Introduction

This chapter is devoted to the dimensionality reduction problem and presents our second contribution.

First, a brief overview of the dimensionality reduction problem is provided. A presentation of the principal component analysis PCA method and related state-ofthe-art of dimentionality reduction are followed in Section 5.2 in order to describe more precisely the framework of this Chapter.

Next, we present our main contributions of this chapter. The two dimensionality reduction models related to GMM and GGMM will be presented, respectively, in Section 5.3 and Section 5.4. These models are proposed to combine a GMM or GGMM with a principal component analysis (PCA) and derive an EM algorithm for estimating the parameters of the arising model. Finally, these dimensionality reduction models are applied to the SR image using the MMSE estimator with the GMM and GGMM models. The results of the SR methods are presented in Section 5.5.

The conclusion is given in Sections 5.6.

State-of-the-art of dimensionality reduced data

An important motivation for dimensionality reduction is that many datasets have the property that all data points lie close to a manifold. This manifold has a much lower dimensionality than the original data space. Dimensionality reduction is therefore the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data. The standard method for dimensionality reduction is the principal component analysis (PCA) [START_REF] Pearson | On lines and planes of closest fit to systems of points in space[END_REF], which was introduced in 1901 by Karl Pearson. Based on this method, [START_REF] Tipping | Probabilistic Principal Component Analysis[END_REF] introduced the probabilistic principal component analysis (PPCA) for the Gaussian distribution. It was extended to the Gaussian Mixture model in [START_REF] Tipping | Mixture of probabilistic principal component analyzers[END_REF] with the mixture of probabilistic principal component analysers (MPPCA) model. Beside that, the authors of [START_REF] Bouveyron | High-dimensional data clustering[END_REF] proposed a dimensionality reduction method for Gaussian mixture model, called High-Dimensional Data Clustering (HDDC) method. It is based on the MPPCA model to replace the affine space in the PCA by the union of finitely many affine spaces using a mixture model of probabilistic PCA.

Principal Component Analysis (PCA) [72]

PCA is a well-known technique for dimensionality reduction. PCA can be defined as the linear projection that minimizes the mean squared distance between the data points and their projection. Based on this definition, given data samples X = {x 1 , ..., x N } in R n with dimensionality n, the classical PCA finds the latent d-dimensional affine space {U t + b : t ∈ R d }, 1 ≤ d ≪ n having smallest squared distance from the samples by minimizing

P (U, b) = N i=1 ∥(U U T -I n )(x i -b)∥ 2 (5.
P (U ) = N i=1 ∥(U U T -I n )y i )∥ 2 , y i = x i -x.
Now we can estimate matrix U with constraint U ∈ St(d, n). To enforce this constraint, we introduce a Lagrange multiplier that we will denote by Λ ∈ R d×d .

The minimization problem to P (U ) with constraint U ∈ St(d, n) become an unconstrained minimization of

L (U, Λ) = N i=1 ∥(U U T -I n )y i )∥ 2 + tr U T U -I d .
By setting the derivative of L (U, Λ) with respect to U equal to zero, this quantity will have a stationary point when

SU = ΛU

where S = N i=1 y i y T i . This implies that a minimizer can be derived explicitly as the matrix Û , whose columns are given by the eigenvectors corresponding to the d largest eigenvalues of the empirical covariance matrix S. This minimizer is not unique, since it holds P (U V ) = P (U ) for any orthogonal matrix V ∈ O(d).

In summary, the formulation of PCA is based on a linear projection of the data onto a subspace of lower dimensionality than the original data space. To apply the PCA for dimensionality reduction, we first find eigenvectors and eigenvalues of matrix S, and then compute the low-dimentional data from original data space using U T (x -b) : x ∈ R n . In the next subsection, we consider a reformulation of PCA, known as probabilistic PCA (PPCA). This formulat shows that PCA can also be expressed as the maximum likelihood solution of a probabilistic latent variable model.

Probabilistic Principal Component Analysis PPCA [79]

The probabilistic PCA model was proposed by Tipping and Bishop in [START_REF] Tipping | Probabilistic Principal Component Analysis[END_REF]. This model is considered as a Gaussian model. Based on PCA's goals, the mapping of latent data to original data can be defined as a linear function of t:

x = U t + b + ε, (5.2.2)
where the latent variables are defined to be independent and Gaussian with a zeromean unit-covariance t ∼ N (0, I). The noise of the model is also Gaussian such that ε ∼ N 0, σ 2 I . Thus the conditional probability distribution over x-space for a given t is x | t ∼ N U t + b, σ 2 I . The probability density function of this distribution is written as

p (x | t) = 1 2πσ 2 exp - 1 2σ 2 ∥x -U t -b∥ 2 .
(5.2.3)

To determine the parameters U, b, σ 2 by using maximum likelihood, we need an expression for the marginal distribution of original variables p (x). The expression of marginal distribution p (x) can be obtained thanks to the sum and product rules of probability, in the form:

p (x) = p (x | t) p (t) dt = (2π) -n/2 Σ -1/2 exp - 1 2 (x -b) T Σ-1 (x -b) , (5.2.4)
where Σ = U U T + σ 2 I is the covariance matrix. Now the parameters U, b, σ 2 can be estimated by using maximum likelihood method. Given a data set X = {x 1 , ..., x N } in R n with n-dimensional, the corresponding log-likelihood function is given by

L P U, b, σ 2 = N i=1 log (p (x)) - N 2 n log (2π) + log Σ + tr Σ-1 S , (5.2.5)
where

S = 1 N N i=1 (x i -b) (x i -b) T (5.2.6)
is the sample covariance matrix of the observed {x i } i=1,...,N . Setting the derivative of the log-likelihood function L with respect to b equal to zero, the bias b = x = 1 N (x 1 + ... + x N ). The maximization of the log-likelihood is more complex with respect to U and σ 2 than b. In [START_REF] Tipping | Probabilistic Principal Component Analysis[END_REF], Tipping and Bishop showed that all of the stationary points of the log-likelihood function can be written as

U M L = W Λ -σ 2 I 1/2 R, ( 5
σ 2 M L = 1 n -d n j=d+1 λ j (5.2.8)
where λ d+1 , ..., λ n are the smallest eigenvalues of S. So that σ 2 M L is the average variance associated over the lost dimensions. In practice, to implement probabilistic PCA, we first compute the eigen-decomposition of S. Then we estimate σ 2 M L from (5.2.8), and finally the matrix U M L from equation (5.2.7).

Mixture of probabilistic principal component analysers [80]

The association of a probabilistic PCA model offers attractive prospects for application to dimensionality reduction. In [START_REF] Tipping | Mixture of probabilistic principal component analyzers[END_REF], Tipping and Bishop introduced a model that consider a mixture distribution in which the components are probabilistic PCA. This is called Mixtures of Probabilistic PCA (MPPCA) model. By using the Gaussian mixture models (GMMs) with the density function , the log-likelihood of observing the data set for such a Gaussian mixture model is:

L = N i=1 log K k=1 w k p (x n | k) , (5.2
(r) i,k = w k p (x i | θ k ) K l=1 w l p (x i | θ l ) , ( 5
(r+1) k = N i=1 α (r) i,k x i N i=1 α (r) i,k (5.2

.11)

The transformation matrix U k , and the noise σ 2 k can be obtained from matrix

S (r) k = 1 w (r+1) k N i=1 α (r) i,k x i -b (r+1) k x i -b (r+1) k T (5.2.12)
in the same way as for a single PPCA model and the covariance matrices of the high-dimensional data are Σk = U k U T k + σ 2 I for all k = 1, ..., K.

High-Dimensional Data Clustering (HDDC) [11]

In [START_REF] Bouveyron | High-dimensional data clustering[END_REF], C. Bouveyron et al. proposed a Gaussian mixture models designed for highdimensional data, which is called High-Dimensional Data Clustering (HDDC). This model combines the ideas of subspace clustering and parsimonious modeling based on the Expectation-Maximization (EM) algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF]. The EM algorithm is used to estimate the parameter of the specific subspace and the intrinsic dimension of each mixture component. Given a high-dimensional dataset {x i } i=1,...,N in R n , we assume that the variables have Gaussian mixture distribution with means μk and covariance matrices Σk for k = 1, ..., K. Then the covariance matrix Σk can be factorized as:

Σk = Q k Λ k Q T k (5.2.13)
where Q k is an orthogonal matrix containing the eigenvectors of Σk and Λ k is a n × n diagonal matrix containing the eigenvalues of Σk . It is further assumed that λ k has the following form:

Λ k =           a k1 0 . . . 0 a kd k 0 0 b k 0 . . . 0 b k           , (5.2.14)
with a kj ≥ b k , j = 1, ..., d k and d k ∈ {1, ..., n -1} for k = 1, ..., K.

We consider the affine subspace E k is defined by the d k eigenvectors associated to the eigenvalues a kj and such that μk ∈ E k . Similarly, the affine subspace

E ⊥ k is such that E k ⊕ E ⊥ k = R n and μk ∈ E ⊥ k . Let P k (x) = Qk QT k (x -μk ) + μk and P k (x) ⊥ = Q k Q T k (x -μk ) + μk be the projection of x on E k and E ⊥ k respectively, where Qk is made of the d k first columns of Q k supplemented by n-d k zero columns and Q k = Q k -Qk .
We thus obtain a re-parameterization of the Gaussian mixture model which

depends on θk = {μ k , a kj , b k , Q k , d k } for all k = 1, ..., K. This mixture model is denoted by [a kj , b k , Q k , d k ]
and its probability density function of the k mixture component is written as: ) for the new parameters can be solved by using the method of Lagrange multipliers. We refer to [START_REF] Bouveyron | High-dimensional data clustering[END_REF] for a demonstration of the following results for each mixture component k = 1, ..., K:

p x | θk = 1 (2π) n/2 |Q k Λ k Q T k | 1/2 exp -1 2 (x -μk ) T Q k Λ k Q T k -1 (x -μk ) = 1 (2π) n/2 |Λ k | 1/2 exp -1 2 (x -μk ) T Q k Λ k Q T k -1 (x -μk ) . ( 5
• The mean is estimated by:

µ (r+1) k = 1 N i=1 α (r) i,k N i=1 α (r) i,k x i .
• The empirical covariance matrix is estimated by:

S (r+1) k = 1 N i=1 α (r) i,k N i=1 α (r) i,k x i -µ (r+1) k x i -µ (
(r+1) k = 1 n -d k   tr S (r+1) k - d k j=1 λ kj   .
The main difference between the HDDC method and the others is that it estimates the d k dimension for each cluster of the model. Within the M step of EM algorithm, the intrinsic dimensions d k , k = 1, ..., Kk are estimated through the scree-test of Cattell [START_REF] Cattell | The scree test for the number of factors[END_REF] which is illustrated in Figure 5.1. The selected d k is the one for which the subsequent eigenvalues differences are smaller than the threshold (dashed line). In this paper, new Gaussian mixture models designed for high-dimensional data are introduced. It is assumed that the intrinsic dimension of each mixture component is much smaller than the one of the original space. In addition, outside the specific subspace of each group, the noise variance is modeled by a single parameter. Additional constraints can be imposed on the parameters within or between the groups in order to obtain further regularized models. This parameterization in the eigenspaces of the mixture components gives rise to an EM-based clustering method, called High-Dimensional Data Clustering (HDDC). Experiments on artificial and real datasets demonstrated the effectiveness of the different models of HDDC compared to classical Gaussian mixture models.

We will analyze and compare these methods with ours in the following section 5.3.2.

?chaptername? 5. Dimensionality reduction is an orthogonal matrix. Then we obtain

V T Σ-1 V = V T ( 1 σ 2 (I n -U U T ) + U Σ -1 U T )V (5.3.10) = 1 σ 2 (I n -V T U U T V ) + V T U Σ -1 U T V. (5.3.11) Since (V T U ) T = U T V = (I d |0), this is equal to V T Σ-1 V = 1 σ 2 0 0 0 I n-d + Σ -1 0 0 0 = Σ -1 0 0 1 σ 2 I n-d (5.3.12)
and the last matrix is invertible.

2. We have to show that

(2π) -d 2 |Σ| -1 2 exp - 1 2σ 2 ∥(I n -U U T )(x -b)∥ 2 - 1 2 (U T (x -b) -µ) T Σ -1 (U T (x -b) -µ) = (2π) -n 2 | Σ| -1 2 exp - 1 2 (x -μ) T Σ-1 (x -μ) = (2π) -n 2 | Σ| -1 2 exp - 1 2 x T Σ-1 x + μT Σ-1 x - 1 2 μT Σ-1 μ .
Straightforward calculation together with the observation that U T ΣU = Σ and hence

U T Σ-1 U = Σ -1 gives 1 2σ 2 ∥(I n -U U T )(x -b)∥ 2 + 1 2 (U T (x -b) -µ) T Σ -1 (U T (x -b) -µ) (5.3.13) = 1 2 x T 1 σ 2 (I n -U U T ) + U Σ -1 U T x (5.3.14) -1 σ 2 b T (I n -U U T ) + (µ T + b T U )Σ -1 U T x (5.3.15) + 1 2 (U T b + µ) T Σ -1 (U T b + µ) + 1 2σ 2 b T (I n -U U T )b (5.3.16) = 1 2 x T Σ-1 x -μT Σ-1 x + 1 2 μT Σ-1 μ. (5.3.17)
Finally, we see by (5.3.12)

that | Σ| -1 = σ -2(n-d) |Σ| -1 .
By Lemma 5.3.1, we can rewrite our objective function F in (5.3.5) with Θ = (μ, Σ) defined by (5.3.8) and (5.3.9) with corresponding indices as

F (U, b, w, Θ) = - N i=1 log K k=1 w k f (x i | θk ) + (n -d) log( √ 2πσ 2 ) (5.3.18) = L(w, Θ|X ) + (n -d) log( √ 2πσ 2 ). (5.3.19)
Up to the constant this is a negative log-likelihood function of a GMM. However, when minimizing this function, we have to take the constraints (5.3.9) and ( 5 5.3.20) where

F(U, b, w, Θ) := L(w, Θ|X ) subject to U k ∈ St(d, n), w ∈ ∆ K , Σ k ∈ SPD(d), (
Σk = 1 σ 2 (I n -U k U T k ) + U k Σ -1 k U T k -1 , μk = Σk U Σ -1 k µ k + b k , k = 1, . . . , K. (5.3.21)
The choice of µ k and b k is redundant. This can be seen as follows, for any µ k and b k , define μk = 0 and bk = μk . Then, it holds 5.3.22) such that F(U, b, w, Θ) = F(U, b, w, Θ). Consequently, in the M-step of the EM algorithm in Section 5.3.3.1 we obtain that the update for the mean µ is given by µ k = 0.

Σk = 1 σ 2 (I n -U k U T k ) + U k Σ -1 k U T k -1 , μk = Σk U Σ -1 k μk + bk , k = 1, . . . , K (
Remark 4 (Different component dimensions). So far the dimension d is the same for all components k = 1, ..., K. But by some simple adjustments, the PCA-GMM model can also be rewritten with

U k ∈ St(d k , n), µ k ∈ R d k and Σ k ∈ SPD(d k )
, where the d k are not necessarily equal for all k. However, to keep the notations as simple as possible, we will restrict our analysis to the case that d k = d for k = 1, ..., K.

Nevertheless, all the results of this chapter can be derived analogously for other choices of d k .

Remark 5 (Learning σ). The function F in (5.3.1), resp. (5.3.5), (5.3.19) is strictly decreasing in σ. Thus it does not make sense to minimize F with respect to σ. However, the function F = F -n-d 2 log(2πσ 2 ) in (5.3.20) can be optimized with respect to σ. To keep the M-step of the EM algorithm simple, we associate to each sum and in the mixture model an own σ k , k = 1, . . . , K such that Σ in (5.3.21) becomes

Σk = 1 σ 2 k (I n -U k U T k ) + U k Σ -1 k U T k -1 . (5.3.23)
In this case, we use the notation σ = (σ k ) K k=1 .

Comparison with state-of-the-art of dimensionality reduced data

There are several relations of the PCA-GMM model to other models proposed in the literature, in particular to mixtures of probabilistic PCAs (MPPCA) [START_REF] Tipping | Mixture of probabilistic principal component analyzers[END_REF], high dimensional data clustering (HDDC) [START_REF] Bouveyron | High-dimensional data clustering[END_REF] and high-dimensional mixture models for unsupervised image denoising (HDMI) [START_REF] Houdard | High-dimensional mixture model for unsupervised image denoising (HDMI)[END_REF]. In the following, we shortly review these methods and comment on similarities and differences to the PCA-GMM model.

For understanding the relation to other models, we first need the following reformulation of the covariance matrices Σ from the PCA-GMM model. We have as in (5.3.12) for matrices Σ of the form (5.3.9) and an orthogonal matrix V = (U | Ũ ) that .3.24) so that

V T ΣV = Σ 0 0 σ 2 I n-d , ( 5
Σ = 1 σ 2 (I n -U U T ) + U Σ -1 U T -1 : U ∈ St(d, n), Σ ∈ SPD(d) (5.3.25) = Q T diag(λ) 0 0 σ 2 I n-d Q : Q ∈ O(n), λ ∈ R d >0 .
(5.3.26)

As outlined in Remark 5, the σ can either be fixed a priori, or optimized within the EM algorithm, as later outlined in Section 5.3.3.1, simultaneously with the other parameters.

In [START_REF] Tipping | Mixture of probabilistic principal component analyzers[END_REF], Tipping and Bishop propose mixture models of probabilistic PCAs (MPPCA), which are GMMs of the form

p(x) = K k=1 w k f (x i |μ k , Σk ), (5.3.27)
where

Σk = U k U T k + σ 2 k I n , U k ∈ St(d k , n).
Here, the parameters σ k are optimized simultaneously with the w k and U k via the EM algorithm. Hence, skipping the index, instead of minimizing over (5.3.26), they minimize over sets of the form

Q T (1 + σ 2 )I d 0 0 σ 2 I n-d Q : Q ∈ O(n) . (5.3.28)
Since this form of the covariance matrices is very restrictive, Bouveyron, Girard and Schmid generalized MPPCA in [START_REF] Bouveyron | High-dimensional data clustering[END_REF] to a model called high dimensional data clustering (HDDC). Again, they minimize a special GMM (5.3.27), but here the covariances are given by

Σk = U k diag(λ k )U T k + σ 2 k I n , U k ∈ St(d k , n), λ k ∈ R d k >0 .
As for the MPPCA, the parameters are optimized via the EM algorithm. For deriving it, it is important that the parameters σ k are not fixed a priori but are 5.3. Contribution: Dimensionality-Reduced Gaussian Distribution: PCA-GMM Model [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] 59

optimized within the EM algorithm. Skipping the index, instead of minimizing over (5.3.26) or (5.3.28), this corresponds to a minimization over

Q T diag(λ) + σ 2 I d 0 0 σ 2 I n-d Q : Q ∈ O(n), λ ∈ R d >0 .
(5.3.29)

In contrast to (5.3.26), where the diagonal values λ are required to be strictly greater than 0, the diagonal values λ + σ 2 in (5.3.29) are automatically strictly greater than σ 2 . Consequently, the PCA-GMM model is more general than HDDC. Note that HDDC model contains the so-called mixture factor analysis [START_REF] Mclachlan | Modelling high-dimensional data by mixtures of factor analyzers[END_REF] as a special case. Here also the alternating expectation conditional maximization algorithm [START_REF] Meng | The EM algorithm-an old folk-song sung to a fast new tune[END_REF] is applicable [START_REF] Zhao | Fast ML estimation for the mixture of factor analyzers via an ECM algorithm[END_REF], which is an improved version of the EM algorithm.

Finally, Houdard, Bouveyron and Delon proposed in [START_REF] Houdard | High-dimensional mixture model for unsupervised image denoising (HDMI)[END_REF] a model selection algorithm for the dimensions d k . For this, they propose a model called HDMI, where the only difference to HDDC is, that σ is a priori fixed. They derive as an intermediate step a corresponding EM algorithm in [45, Proposition 2]. Unfortunately, the M-step only ensures that λ > -σ 2 1 d and not λ > 0, such that the calculations appear to be not fully correct. However, the final model selection algorithm again ensures that λ > 0 such that this seems not to be a problem in [START_REF] Houdard | High-dimensional mixture model for unsupervised image denoising (HDMI)[END_REF].

Minimization Algorithm

We propose to minimize F in (5.3.20) based on the EM algorithm, where we have to take the special structure of μk ∈ R n and Σk ∈ SPD(n) in (5.3.21) into account to work indeed in the lower d-dimensional space. This requires the solution of a special inner minimization problem within the M-Step of the EM algorithm. We describe the EM algorithm for our PCA-GMM model in Subsection 5.3.3.1. In particular, we will see that the M-Step of the algorithm requires the minimization of functions G k (U, b), k = 1, . . . , K, of the same structure. We prove that these functions have indeed a global minimizer. In particular, these functions do not depend on the large number of input data x i , i = 1, . . . , N . Therefore it turns out that the E-step of the algorithm is the most time consuming one. We propose to find at least a local minimizer of G by the (inertial) proximal alternating linearized minimization (PALM) in Section 5.3.3.2 and provide convergence results.

EM Algorithm for PCA-GMM

For our setting, we obtain a special EM algorithm described in Algorithm 2. Note that the E-Step of Algorithm 2 requires only the mean and covariance matrix in θ (r) k , k = 1, . . . , K with respect to the smaller space R d .

A convergence analysis of the EM algorithm via Kullback-Leibler proximal point algorithms was given in [START_REF] Chrétien | Kullback proximal algorithms for maximumlikelihood estimation[END_REF][START_REF] Chrétien | On EM algorithms and their proximal generalizations[END_REF], see also [START_REF] Laus | Statistical Analysis and Optimal Transport for Euclidean and Manifold-Valued Data[END_REF] for a nice review. The authors showed that the objective function decreases for the iterates of the algorithm. Hence we obtain the following corollary.

?chaptername? 5. Dimensionality reduction Algorithm 2 EM Algorithm for PCA reduced Mixture Models Input: X = (x 1 , ..., x N ) ∈ R n,N , initialization U (0) , b (0) , w (0) , θ (0) = (µ (0) , Σ (0) ). for r = 0, 1, ... do E-Step: Proposition 1. Assume that n + 1 of the points x i , i = 1, ..., N are affinely independent.

For k = 1, ..., K and i = 1, . . . , N compute α (r) i,k = w (r) k f (x i | θ(r) k ) K j=1 w (r) j f (x i | θ(r) j ) (5.3.30) = w (r) k (σ (r) k ) n-d exp -1 2(σ (r) k ) 2 ∥(I n -U (r) k (U (r) k ) T )y (r) i,k ∥ 2 f (U (r) k ) T y (r) i,k |θ (r) k K j=1 w (r) j (σ (r) j ) n-d exp -1 2(σ (r) j ) 2 ∥(I n -U (r) j (U (r) j ) T )y (r) i,k ∥ 2 f (U (r) j ) T y (r) i,k |θ (r) j , (5.3.31) y (r) i,k = x i -b (r) k . (5.3.32) M-Step: For k = 1, ..., K compute w (r+1) k = 1 N N i=1 α (r) i,k , (5.3.33) (U (r+1) k , b (r+1) k , σ (r+1) k , θ (r+1)
k ) = arg max U,b,µ,Σ N i=1 α (r) ik log(f (x i | θk )) (5.3.34) subject to U k ∈ St(d, n), Σ k ∈ SPD(d) (5.
Further, let f be the Gaussian density function (4.1.6) and where Note that w in the proposition is defined in another way than in the first Mstep, more precisely, the factor 1 N is skipped. Before presenting the proof of the proposition, we give the following remark. Remark 6. By definition of C in Proposition 1 we have that

α i ∈ R ≥0 , i = 1, . . . , N . i) For fixed σ 2 , a solution of arg max U,b,µ,Σ N i=1 α i log(f (x i | θ)) (5.
m = N i=1 α i x i , C = N i=1 α i x i x T i , w = N i=1 α i , S = C - 1 w mm T , ( 5
S = N i=1 α i (x i -1 w m)(x i -1 w m) T . ( 5
α i U T (x i -b) = 1 w (U T m -wU T b), (5.3.45) Σ = 1 w N i=1 α i U T (x i -b) -µ U T (x i -b) -µ T (5.3.46) = 1 w N i=1 α i U T (x i - 1 w m) U T (x i - 1 w m) T = 1 w U T SU. (5.3.47) Σk = ΣH,k ΣHL,k ( ΣHL,k ) T ΣL,k
. With parameters w k , μk , Σk , k = 1, . . . , K, the high-resolution patches can be estimated as in equation (4.2.22). Finally, the highresolution image is reconstructed by applying the reconstruction of high-resolution image step by patch averaging as in 3, Subsection 4.1.1. We will demonstrate the performance of PCA-GMM method by two-and three-dimensional examples in Subsection 5.5.1.

Contribution: Dimensionality-Reduced Generalized

Gaussian Distribution: PCA-GGMM Model [START_REF] Nguyen | Image superresolution with PCA Reduced generalized Gaussian mixture models[END_REF] This section aims to present our second contribution for dimensionality reduction topic. This contribution develops the PCA-GMM model by incorporate a dimensionality reduction within mixture models of generalized Gaussian distributions and derive an algorithm for estimating its parameters. Additionally to these methodical improvements, we apply our method to three-dimensional realworld images showing material microstructures. This contribution was submitted for the Inverse Problems in Imaging journal [START_REF] Nguyen | Image superresolution with PCA Reduced generalized Gaussian mixture models[END_REF] in 2022. We will discuss it in the following.

Combining PCA with generalized Gaussian distribution

In the following, we consider the samples x 1 , ..., x N ∈ R n , which are approximately located in a d-dimensional subspace of R n . Then, our objective is to find simultaneously the subspace

{U t + b : t ∈ R d }, U ∈ St(d, n), b ∈ R n that
contains the samples x i and a generalized Gaussian distribution GG(µ, Σ, β) with density function q(•|µ, Σ, β). For this, we propose a model that combines the PCA and the generalized Gaussian model of the low-dimensional space. Based on the idea of the PCA-GMM model, we are motivated to minimize the following function

ℓ PCA (U, b, µ, Σ, β) := N i=1 1 2σ 2 ∥(U U T -I)(x i -b)∥ 2 -log(q(U T (x i -b)|µ, Σ, β)),
(5.4.1) which is a weighted sum of a PCA term and the negative log-likelihood function of q within the lower dimensional subspace.

The following proposition gives a general property of the function ℓ P CA for the generalized Gaussian distribution. The function ℓ PCA is up to a constant the negative log-likelihood function of a distribution P θ with θ = (U, b, µ, Σ, β). Proposition 2. Let GG(µ, Σ, β) be a generalized Gaussian distribution with a density function q(•|µ, Σ, β). The function ℓ PCA (U, b, µ, Σ, β) is up to a constant 5.4. Contribution: Dimensionality-Reduced Generalized Gaussian Distribution: PCA-GGMM Model [START_REF] Nguyen | Image superresolution with PCA Reduced generalized Gaussian mixture models[END_REF] 69

the negative log-likelihood function of a probability density

p(x|U, b, µ, Σ, β) := 1 (2πσ 2 ) (n-d)/2 exp(-1 2σ 2 ∥(U U T -I)(x-b)∥ 2 )q(U T (x i -b)|µ, Σ, β).
(5.4.

2) The distribution P θ with θ = (U, b, µ, Σ, β) corresponding to this density is given by

P θ = r # (GG(µ, Σ, β) ⊗ N (0, σ 2 I n-d )), r(x) = V x + b,
where V = (U | Ũ ) is an orthogonal matrix, ⊗ denotes the product measure and r # P is the push-forward measure of P under r.

?proofname? Using that U T U = I d and Ũ T Ũ = I n-d , we have that the distribution GG(µ, Σ, β) ⊗ N (0, σ 2 I n-d ) has at x = (x 1 , x 2 ) ∈ R n the density 1 (2πσ 2 ) (n-d)/2 exp(-1 2σ 2 ∥ Ũ T Ũ x 2 ∥ 2 )q(U T U x 1 |µ, Σ, β)
As U T Ũ = 0 and Ũ T U = 0, we obtain, that this is equal to

1 (2πσ 2 ) (n-d)/2 exp(-1 2σ 2 ∥ Ũ T (U x 1 + Ũ x 2 )∥ 2 )q(U T (U x 1 + Ũ x 2 )|µ, Σ , β) (5.4.3)
= 1 (2πσ 2 ) (n-d)/2 exp(-1 2σ 2 ∥ Ũ T V x∥ 2 )q(U T V x|µ, Σ, β) (5.4.4)
Using the change of variables formula for push-forward measures, we obtain that P θ has the density

1 (2πσ 2 ) (n-d)/2 exp(-1 2σ 2 ∥ Ũ T V r -1 (x)∥ 2 )q(U T V r -1 (x)|µ, Σ, β)|det(∇r -1 (x))|
As it holds r -1 (x) = V T (x -b) and as ∇r -1 (x) = V T is an orthogonal matrix, we obtain that this is equal to

1 (2πσ 2 ) (n-d)/2 exp(-1 2σ 2 ∥ Ũ T (x -b)∥ 2 )q(U T (x -b)|µ, Σ, β) (5.4.5)
where we used V V T = I n . Finally, the fact that the mapping x → Ũ x is an isometry and that

I -U U T = Ũ Ũ T imply that ∥ Ũ T (x -b)∥ 2 = ∥ Ũ Ũ T (x -b)∥ 2 = ∥(U U T -I)(x -b)∥ 2
such that the density (5.4.5) coincides with p(x|U, b, µ, Σ, β). Taking the negative logarithm and subtracting n-d 2 log(2πσ 2 ) shows that the negative log-likelihood function of p(•|U, b, µ, Σ, β) coincides up to a constant with ℓ PCA (U, b, µ, Σ, β).

Due to its relationship to the PCA, we call the probability distribution p(•|U, b, µ, Σ, β) a PCA-reduced generalized Gaussian distribution.

Remark 9 (PCA-GMM Model [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF]). In particular, our PCA-reduced Gaussian distribution (PCA-GMM model in the case of mixture model) is again a Gaussian distribution with a special structured covariance matrix. Using this result, we have derived an EM algorithm for PCA-reduced Gaussian mixture models, which showed a significant speed-up compared to a high-dimensional GMM. However, for generalized Gaussian distribution, the PCA-reduced counterpart is not a generalized Gaussian distribution such that the analysis of [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] is no longer applicable.

In the following subsection, we introduce a method for estimating the parameters of mixture models for the PCA-reduced generalized Gaussian distribution. We call this mixture model a PCA-GGMM model, which has density function as

F (x | w, Θ) = K k=1 w k p (x | θ k ) (5.4.6)
where the weights w k are non-negative and sum up to one, Θ = {θ k } k=1,...,K with

θ k = (U k , b k , µ k , Σ k , β k ) and p (x | θ k ) is defined as (5.4.2).

Weighted Maximum Likelihood Estimation

The PCA-GGMM model is estimated based on EM Algorithm 1. For each k mixture cluster of the PCA-GGMM model, the weighted maximum likelihood estimation (4.2.5) is estimated for PCA-reduced generalized Gaussian distribution.

For readability, we consider the estimation for the the weighted maximum likelihood estimation with parameter θ = (U, b, µ, Σ, β) instead of θ k .

To estimate the parameters of a PCA-reduced generalized Gaussian distribution from samples x 1 , ..., x N with weights α 1 , ..., α N , our objective is to minimize the weighted negative log-likelihood function

N i=1 α i log(p(x i |U, b, µ, Σ, β)).
By Proposition 2 this is equivalent to minimizing the function

ℓ PCA,α (U, b, µ, Σ, β) := N i=1 α i 1 2σ 2 ∥(U U T -I)(x i -b)∥ 2 -log(q(U T (x i -b)|µ, Σ, β)) ,
which is up to the weights α the same function as ℓ PCA from (5.4.1). Unfortunately, we cannot minimize ℓ PCA,α directly. Instead, we apply a technique called halfquadratic splitting [START_REF] Geman | Nonlinear image recovery with half-quadratic regularization[END_REF]. That is, instead of considering ℓ PCA,α (U, b, µ, Σ, β), we minimize for some large η > 0 the function i.e., we replace U T (x i -b) in the second summand by z i and penalize the squared distance of U T (x i -b) and z i . Thus, for η → ∞, the minimizers of H σ,η converge to the minimizers of ℓ PCA,α . Now, we minimize H σ,η iteratively with respect to (U, b), z and (µ, Σ, β), i.e., we generate a sequence (U (r) , b (r) , µ (r) , Σ (r) , β (r) ) r by iteration 1) , Σ (r-1) , β (r-1) ))

H σ,η (z, U, b, µ, Σ, β) := N i=1 α i 1 2σ 2 ∥(U U T -I)(x i -b)∥ 2 +η∥U T (x i -b)-z i ∥ 2 -log(q(z i |µ, Σ, β)) ,
U (r) ∈ arg min U ∈St(d,n) N i=1 α i 1 2σ 2 ∥(U U T -I)(x i -b (r-1) )∥ 2 + η∥U T (x i -b (r-1) ) -z (r-1) i ∥ 2 , (5.4.7)
b (r) ∈ arg min b∈R n N i=1 α i 1 2σ 2 ∥(U (r) (U (r) ) T -I)(x i -b)∥ 2 + η∥(U (r) ) T (x i -b) -z (r-1) i ∥ 2 , (5.4.8)
z (r) ∈ arg min z=(z 1 ,...,z N )∈R d,N N i=1 α i η∥(U (r) ) T (x i -b (r) ) -z i ∥ 2 -log(q(z i |µ (r-
(5.4.9)

and finally

(µ (r) , Σ (r) , β (r) ) ∈ arg max µ∈R d ,Σ∈SPD (d),β>0 N i=1 α i log(q(z (r) i |µ, Σ, β)) (5.4.10)
The final step (5.4.10) is the weighted maximum likelihood estimation of the generalized Gaussian distribution q, which was discussed in the previous section.

It remains to solve the first three steps.

Solving (5.4.7) by Uzawas' Algorithm. Using that for any y ∈ R n and U ∈ St(d, n) it holds that

∥(U U T -I)y∥ 2 = y T (U U T -I) 2 y = y T (U U T U U T -2U U T + I)y = y T (U U T -2U U T + I)y (5.4.11) = y T (I -U U T )y = ∥y∥ 2 -∥U T y∥ 2 , (5.4.12)
the optimization problem in (5.4.7) reads as

arg min U ∈St(d,n) N i=1 α i - 1 2σ 2 ∥U T (x i -b)∥ 2 + η∥U T (x i -b) -z i ∥ 2
Now, we transform the problem in an unconstrained problem by considering the Lagrangian function

L(U, Λ) = N i=1 α i - 1 2σ 2 ∥U T (x i -b)∥ 2 + η∥U T (x i -b) -z i ∥ 2 + tr(Λ(U T U -I)).
?chaptername? 5. Dimensionality reduction Then, the solution of (5.4.7) is equivalent to solving the saddle-point problem arg min

U ∈R n,d max Λ∈R d,d
L(U, Λ).

To solve this saddle-point problem, we use the Uzawas' algorithm [START_REF] Arrow | Studies in linear and nonlinear programming[END_REF], which consists of the following two steps: First, we minimize L with respect to U . Second, we perform a gradient ascent step with respect to Λ.

The second step is given by

Λ (r+1) = Λ (r) + ρ((U (r+1) ) T U (r+1) -I)
In the case of our specific Lagrangian L, the second step is given by the following proposition.

Proposition 3. Let Λ be fixed. Then, any minimizer of L(U, Λ) solves the Sylvester equation

(η -1 2σ 2 ) N i=1 α i (x i -b)(x i -b) T U + U Λ = η N i=1 α i (x i -b)z T i .
?proofname? We aim to minimize the Lagrangian function

L(U, Λ) = N i=1 α i - 1 2σ 2 ∥U T (x i -b)∥ 2 + η∥U T (x i -b) -z i ∥ 2 + tr(Λ(U T U -I)).
for fixed Λ with respect to U by setting the gradient to zero. Since it holds by [START_REF] Petersen | The Matrix Cookbook[END_REF] that

∂a T U U T b ∂U = U T (ab T + ba T ) and ∂tr(ΛU T U ) ∂U = ∂tr(ΛU T U ) ∂(U T U ) ∂U T U ) ∂U = 2Λ T U T ,
Thus, it holds that

∇ U L(U, Λ) = 2(η -1 2σ 2 ) N i=1 α i (x i -b)(x i -b) T U -2η N i=1 α i (x i -b)z T i + 2U Λ.
Setting the gradient to zero, it shows that U is a solution of

(η -1 2σ 2 ) N i=1 α i (x i -b)(x i -b) T U + U Λ = η N i=1 α i (x i -b)z T i .
This finishes the proof.

Solving (5.4.8). Setting the gradient of the objective function to zero leads to 5.4. Contribution: Dimensionality-Reduced Generalized Gaussian Distribution: PCA-GGMM Model [START_REF] Nguyen | Image superresolution with PCA Reduced generalized Gaussian mixture models[END_REF] 73

N i=1 α i 1 2σ 2 I -U U T + ηU U T (b -x i ) -ηU z i = 0 (5.4.13) which is equivalent to b = N i=1 α i x i + 1 2σ 2 I -U U T + ηU U T -1 η N i=1 α i U z i N i=1 α i . (5.4.14)
Solving (5.4.9) using a Gradient Descent. Due to the sum-structure of problem (5.4.9), the solution decouples into 1) , Σ (r-1) , β (r-1) )). (5.4.15) This is now a d-dimensional optimization problem, which (owing to its low dimension) can be solved efficiently via a gradient descent scheme.

z (r) i ∈ arg min z i ∈R d η∥(U (r) ) T (x i -b (r) ) -z i ∥ 2 -log(q(z i |µ (r-
Remark 10 (Differentiability of the objective). In the case of generalized Gaussian distributions, the function g is given by g(x) = exp(-1 2 x β ), which is differentiable in {0, +∞}. Thus, the objective function in (5.4.15) is differentiable for any z i ̸ = µ (r-1) . In the case β ≥ 1, the derivative of g can be continuously extended to 0. In this case, the objective function in (5.4.15) is differentiable.

The reduction in dimensionality of the PCA-GGMM model reduces the execution time of the E-step, but the M-step is slower than the GGMM due to its higher complexity. However, all computations in the M-step are on d-dimensional optimization problems, and the FP iteration is implemented on d-dimensional data. This limits the numerical instability problem to estimate the generalized Gaussian mixture model of the original data using the FP-EM algorithm. Moreover, the PCA-reduced generalized Gaussian distribution can be generalized to the PCAreduced elliptical distribution by combining PCA with the elliptical distribution E n (µ, Σ, g) with the density function q(•|µ, Σ, g). We denote p(•|U, b, µ, Σ, g) as a probability density function of the PCA-reduced elliptical distribution. By using the half-quadratic splitting technique, we can apply the iterations (5.4.7), (5.4.8), (5.4.9) and (5.4.10) with q(•|µ, Σ, g) instead of q(•|µ, Σ, β) to estimate the parameters of a PCA-reduced elliptical distribution p(•|U, b, µ, Σ, g).

PCA-GGMM model with application in super-resolution

In the PCA-GGMM model, the latent data U T (x i -b) have a d-dimensional generalized Gaussian mixture distribution with parameters θ k = {µ k , Σ k , β k } for all k = 1, ..., K. Due to the proposition 4 and the definition of the affine subspace U t + b : t ∈ R d , the high-dimensional variables x 1 , ..., x N ∈ R n have a generalized Gaussian mixture distribution with parameters θ k = µ k , Σ k , β k for Choice of σ and K. The PCA-GMM model and PCA-GGMM model depend heavily on the choice of the parameter σ. As pointed out in Subsection 5.3.3.1, this parameter could be learned from the data. However, the forward model for the lowresolution images X L = AX H + ε for some (unknown) super-resolution operator A, the high-resolution image X H and noise ε suggest to choose the σ according to the standard deviation of the noise ε. Note, that in our experiments, the low-resolution images are artificially generated by applying a downsampling operator and adding some noise. Consequently, the standard deviation of ε is known. Nevertheless, if the noise level is unknown, it could be estimated very accurately from the data based on homogeneous area detection as done, e.g., in [START_REF] Hasannasab | Alternatives to the EM algorithm for ML estimation of location, scatter matrix, and degree of freedom of the student t distribution[END_REF][START_REF] Sutour | Estimation of the noise level function based on a nonparametric detection of homogeneous image regions[END_REF].

In practice, it can be unstable to estimate the standard deviation of the noise within the optimization of the mixture model, since this requires that the image patches belong exactly (and not only approximately) to a dimensionality reduced GMM and GGMM with K components, which is an unrealistic assumption. Therefore, it can be beneficial and quite more accurate to estimate the standard deviation of the noise a priori. In particular, if the standard deviation of the noise is known, fixing σ can be the better approach.

Note that the noise with standard deviation σ within the super-resolution model does not necessarily imply that the eigenvalues of the covariance matrices in the mixture model are greater than or equal to σ2 (which is assumed for HDDC), since the noise is only applied to the low-resolution images.

Also the number of components K of the mixture models can have a large impact on the results. For super-resolution, a detailed comparison of the prediction quality for different choices of K was done by Sandeep and Jacob in [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF]. They observed that the benefit of taking more than 100 components in the GMM is usually very small. Therefore, we take K = 100 components for all mixture model in our numerical examples.

PCA-GMM model

All implementations in this subsection were done in Python and Tensorflow and they can be parallelized on a GPU. We run all our experiments on a Lenovo ThinkStation with Intel i7-8700 6-Core processor with 32GB RAM and NVIDIA GeForce GTX-2060 Super GPU. The code is available online 1 .

For the implementation of PALM and iPALM, we use the implementation framework from [START_REF] Hertrich | Inertial stochastic PALM (iSPALM) and applications in machine learning[END_REF] 2 . As suggested in [START_REF] Pock | Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems[END_REF] we set the extrapolation factors γ ?chaptername? 5. Dimensionality reduction Comparison of the computation times. Note that there already exist implementations of HDDC by some of the authors of [START_REF] Bouveyron | High-dimensional data clustering[END_REF]. However, to provide a fair comparison of the execution times, we reimplement the EM algorithm for HDDC in Python and Tensorflow, such that it supports GPU parallelization. Further, note that we compute the updates of α, m and C simultaneously to the E-step such that the corresponding execution time is contained in the E-step, even though the updates technically belong to the M-step. This has the advantage that we have to iterate only once over the whole data set and enables a better parallelization. We implemented this optimization of the order of computation for all of the models (GMM, PCA-GMM and HDDC) analogously.

2D-Data. For estimating the parameters of the mixture models, we use the upper left quarter of the image as in the top row of Figure 4.1. As ground truth for the reconstruction we use the whole images as in the bottom row. The images in the left and middle columns are the middle slices of the material data "FS" and "SiC Diamonds". The high resolution images have a size of 2560×2560. The right column contains the goldhill image, which has the size 512 × 512.

We estimate the parameters of a GMM and of our PCA-GMM as described in the previous sections. First, we fix the parameter σ in Algorithm 2 as the standard deviation of the noise on the low dimensional image (i.e. σ = 0.02). Second, we consider the case when σ is learned from the data and finally we compare our results with HDDC [START_REF] Bouveyron | High-dimensional data clustering[END_REF]. Each mixture model has K = 100 classes. We use the magnification factors q ∈ {2, 4} and the patch size τ = 4 for the low resolution patches. This corresponds to a patch size of qτ = 8 or qτ = 16 respectively for the high resolution images. For the material images, this leads to N ≈ 400000 patches for q = 2 and N ≈ 100000 for q = 4. Using the goldhill image, we get N ≈ 15000 patches for q = 2 and N ≈ 3700 patches for q = 4. We reduce the dimension of the pairs of high and low resolution patches from n = (q 2 +1)τ 2 = 80 or n = (q 2 +1)τ 2 = 272 respectively to d for d ∈ {4, 8, 12, 16, 20}. After estimating the mixture models, we use the reconstruction method from [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] as described in the previous section to reconstruct the ground truth from the artificially downsampled images. The resulting PSNRs are given in Table 5.1. As a reference we also measure the PSNR of the bicubic interpolation. The average execution times for one E-step and one Mstep are given in Table 5.2. Figure 5.2 shows some small areas of the high resolution images, low resolution images and the corresponding reconstructions for GMMs and PCA-GMM with d = 12 and d = 20. The result with d = 12 for PCA-GMM is already almost as good as GMM, whereas the dimension of the patches was reduced by a factor between 4 and 22 (depending on the case). Further, we observed that the dimensionality reduction reduces the execution time of the E-step significantly. On the other hand, the execution time of the M-step is larger than those in the GMM for all dimension reduced models due its higher complexity. Comparing the different dimensionality reduced models, we observe that the PCA-GMM with fixed σ gives significantly better results than the other models, while HDDC achieves the fastest M-step due to the closed-form updates. However, compared to the execution time of the E-step, this advantage seems to be negligible for large data sets as, e.g., the patches from the FS and SiC Diamonds image. PCA-GMM, σ = 0.02 Figure 5.3 shows a histogram of the eigenvalues of the covariance matrices Σ k , k = 1, ..., K of the PCA-GMM model with fixed σ = 0.02 for the FS and SiC Diamonds image with magnification q = 4. We observe, that for the SiC Diamonds image a significant amount of eigenvalues are smaller than σ 2 = 4 • 10 -4 which is not possible within a HDDC model [START_REF] Bouveyron | High-dimensional data clustering[END_REF]. For the FS image, the eigenvalues are mostly greater than σ 2 . 3D-Data. In the following, we present the same experiments as in the 2D-case but with 3D-data. For this experiment, we crop a 600 × 600 × 600 image from the material images "FS" and "SiC Diamonds". For the estimation of the mixture model, we use the upper front left 300 × 300 × 300 part of the images and crop randomly N = 1000000 patches.

Again, we estimate the parameters of a GMM and a PCA-GMM with K = 100 classes and fixed σ = 0.02 as described in the previous sections. Since we have seen in the 2D examples that the results of PCA-GMMs with learned σ and HDDC are ?tablename? 5.2: Average execution time (in seconds) for the E-step and M-step in the EM algorithm for estimating the parameters of the mixture models.

?figurename? 5.2: Reconstructions of 2D low resolution images. The columns from left to right correspond to the following materials images: Diamond with q = 2, Diamond with q = 4, FS with q = 2, FS with q = 4. similar, we compare our 3D results just with HDDC. As magnification factor, we use q = 2. For the low resolution image we use τ ×τ ×τ -patches with patch size τ = 4 and for the high resolution image we use a patch size of qτ = 8. We reduce the dimension of the pairs of high and low resolution patches from n = (q 3 + 1)τ 3 = 576 to d for d ∈ {20, 40, 60}. After estimating the mixture models, we use the reconstruction method from [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] as described in the previous paragraph to reconstruct the ground truth from of the artificially downsampled images. The resulting PSNRs are given in Table 5.3 and the average execution times of one E-step and one M-step are given in Table 5

PCA-GGMM

In the second subsection, we focus on the dimensionality reduction task with our PCA-GGMM model for application in super-resolution. We compare our model with the PCA-GMM [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] on 2D artificially images as in the subsection 4.2.3 and the two-and three-dimensional material images: FS", "SiC Diamonds" with the zooming ratio q = 2, 4.

As in the previous experiments with MMSE-GGMM, we use the patch size τ = 4 for the low resolution and qτ = 8 (or qτ = 16) for high resolution images. For the material images with size 2500 × 2500 of HR, this leads to N ≈ 400000 patches for q = 2 and N ≈ 100000 patches for q = 4. These patches will be used for the training step of the PCA-GGMM model. In this step, we use K = 100 components for the mixture model. Each training patch V of size D = τ 2 q 2 + 1 = 80 or D = τ 2 q 2 + 1 = 272 is reduced to d low dimensional for d ∈ {4, 8, 12, 16, 20}. The PSNR values in Tables 5.5, 5.6 and 5.7 demonstrate that our PCA-GGMM model produces results almost as good as MMSE-GGMM and even slightly better than MMSE-GMM.

For the 3D material data, we crop a 600 × 600 × 600 image from the material images FS and SiC Diamonds. Training data are taken from the upper left part of the 3D image 300 × 300 × 300. We use the magnification factor q = 2, the lowresolution patch size τ = 4, and the high-resolution patch size qτ = 8. Thus, the dimension of the pair high and low resolution patches is D = τ 3 q 3 + 1 = 576. These high-dimensional data are reduced to d for d ∈ {20, 40, 60} in the PCA-GGMM model. Table 5.8 shows the PSNR values of the MMSE method for the GMM, LMM, GGMM model and the PCA-GMM, PCA-GGMM model.

Conclusion

In this chapter, we presented two new algorithms to perform image super-resolution. In the first contribution, we added a dimensionality reduction step within the ?tablename? 5.8: PSNRs of the reconstructions of 3D images using MMSE method and PCA-GGMM with q = 2.

?chaptername? 5. Dimensionality reduction GMM model using PCA on patches. The new variational model, called PCA-GMM is of interest on its own, and can be also applied for other tasks. We solved our PCA-GMM model by an EM algorithm with the usual decreasing guarantees for the objective if the E-step and M-step can be performed exactly, see Corollary 1. However, our M-step requires to solve a non-convex constrained minimization problem. Here we propose to use the PALM algorithm and prove that all assumptions for the convergence of the sequence of iterates to a critical point required by [START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF] are fulfilled, see Corollary 2. Our algorithm has the advantage that the M-step is cheap in relation to the E-step since it does not rely on the large numbers of samples in the inner iterations. We have demonstrated the efficiency of the new model by numerical examples, in the case of 2D and 3D images. They confirm that PCA-GMM is an efficient way of reducing the dimension of the patches, while keeping almost the same quality of the results than with a GMM algorithm. This dimensionality reduction is of the utmost importance when dealing with 3D images, where the size of the data gets very large.

The second contribution of this chapter, called PCA-GGMM, incorporates the PCA model with the GGMM model on the low-dimensional data. Experiments with 2D, 3D synthetic, and real material images demonstrated the effectiveness of the PCA-GGMM models. They showed that our model gives a similar quality for the results as MMSE-GGMM and is better than the PCA-GMM model of [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF]. Furthermore, our method, which estimates the parameters of the PCA-reduced generalized model, could be extended to the case of elliptical distributions such as the Student-t distribution or any other one.

In recent years, many deep learning (DL)-based SISR methods have enabled many advances in super-resolution topic. We are aware of the effectiveness of deep learning approaches for super-resolution, e.g. [START_REF] Dong | Image super-resolution using deep convolutional networks[END_REF][START_REF] Lim | Enhanced deep residual networks for single image super-resolution[END_REF][START_REF] Zhang | Residual dense network for image super-resolution[END_REF][START_REF] Ledig | Photo-realistic single image superresolution using a generative adversarial network[END_REF]. In the next chapter, we will consider some deep learning approaches for super-resolution with high magnification factor and high-dimensional data.

?chaptername? 6

A generative adversarial network with contrast change challenge This chapter focuses on the SR method for the contrast change problem of the material image in a framework of generative adversarial networks (GANs), which has made many breakthroughs in image processing recently.

In particular, Sections 6.1, 6.2 provide a brief overview of the tools used. A presentation of convolutional neural networks (CNNs), and particularly of GANs, is followed by a state-of-the-art on GANs in order to describe more precisely the framework of this chapter.

Then in Section 6.3, a RDGAN-contrast method is proposed considering a generator network including two sub-networks in a GAN context. The first generator sub-network deals with the contrast change problem, and the second one focuses on the reconstruction of the HR image.

Convolutional neural network (CNN) for SISR

Recently, convolutional neural network (CNN) based SR methods have achieved significant improvements. Convolutional neural networks are a particular class of neural network designed for image processing. The first convolutional network ?chaptername? 6. A generative adversarial network with contrast change challenge was proposed by LeCun et al. in 1989 [55]. The authors successfully applied backpropagation to train the convolution kernel coefficients to identify and recognize patterns within a series of handwritten zip codes. Specifically, the CNN architectures became popular following the success of the work by Krizhevsky et al. [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF].

Convolutional neural network (CNN)

Convolutional neural networks (CNNs) are distinguished from other neural networks by their superior performance with image, speech, or audio signal inputs. CNNs allow the relationship between input and output variables to be modeled efficiently by composing several layers. They have three main types of layers, which are convolutional layers, pooling layers and fully-connected (FC) layers. In one layer, each neuron is connected to all neurons in the next layer. The output of the neuron is defined by a weighted sum of all inputs received from the previous layer. This weighted sum is then passed through an activation function to produce the output for the next layer. In this section, we will detail the three main types of layers and the activation functions.

Activation functions

Activation function is a node that is placed at the end or between neural networks. This function aim to decide whether a neuron's response would fire or not.This means that if the output of any individual neuron exceeds a threshold value, that node will be activated, sending data to the next layer of the network. Otherwise, no data is transmitted to the next layer. There is a wide variety of different activation functions that can be used in neural networks. For the CNNs, we mention three main activation functions: sigmoid function, softmax function and rectified linear unit (ReLU) function.

Sigmoid function

The logistic sigmoid function for x ∈ R is defined by

g(x) = 1 1 + exp (-x)
.

Sigmoid function has a return value in the range 0 to 1 and is differentiable, which is useful when training a network. However, this function becomes bad for a reliable training neural network because of the problem of gradient degeneration to zero if the network is too deep. Therefore, the sigmoid function can cause a neural network to be blocked at training time.

Softmax function

The softmax function is known as the normalized exponential function and can be regarded as a multiclass generalization of the logistic sigmoid function. The softmax function for a vector x = (x 1 , x 2 , ..., x K ) ∈ R K is defined when K ≥ 1 by the formula

g(x i ) = exp(x i ) K j=1 exp(x)
fori = 1, ..., K.

Rectified Linear Unit (ReLU) function ReLU function is the most widely used activation function in neural networks today. ReLU is simply a max operation between an input and 0 g(x) = max(0, x).

This function has less vanishing gradient problem than the sigmoid activation function, since the gradients will not become zero at least in one direction.

Convolutional layers

The convolutional network starts with the convolutional layers. These can be followed by additional convolutional layers or pooling layers, with the fullyconnected layer being the final layer.

Convolutional layer

The convolutional layer is the core building block of CNN and it accounts for the majority of the computation. The main objective of convolution is to extract features of different complexity from the input image. The extraction of these features is adapted to the problem considered by learning the weights of each convolutional layer. It is the automatic learning of weights that makes convolutional networks very efficient. The objective of these layers is to search for all the features of the input images by convolutional filtering. Thus, the resulting feature map can be seen as a filter that indicates where the features of interest are located in the image. Let us assume that the input will be a gray image, which is made up of a matrix of pixels in two-dimensional (2D). This means that the input will have two dimensions -height, width. The feature detector is a 2D array of weights that represents part of the image. The filter is then applied to an area of the image, and a dot product between the filter and the input pixels is calculated. This dot product is then passed into an output array. The filter then shifts by a stride, repeating the process until the kernel has scanned the entire image.

Pooling layer

The pooling layer is used to reduce the spatial dimensions of the data by combining the outputs of neuron clusters at one layer into a single neuron in the next layer. Two types of pooling used are:

• Max pooling: It selects the pixel with the maximum value for each neighbourhood to send to the output array (Figure 6.1). In this way, it keeps the strongest features. This approach tends to be used more often compared to average pooling.

the ground truth high-resolution image as X H . The bicubic-interpolated image is convolved by learned non-linear filters in different layers, merged to reconstruct the high-resolution image.

?figurename? 6.3: Sketch of the SRCNN architecture [START_REF] Dong | Learning a deep convolutional network for image super-resolution[END_REF].

Patch extraction and representation This step extracts the overlapping patches from the interpolated image X L and represents each patch as a highdimensional vector. The first layer performs a standard convolution with a ReLU activation function on the filter responses and is expressed as an operation F 1.:

F 1 (X L ) = max(0, W 1 * X L + B 1 ) (6.1.1)
where W 1 and B 1 represent the filters and biases, respectively, and * denotes the convolution operation. Here, W 1 corresponds to n 1 filters of support c × f 1 × f 1 , where c is the number of channels in the input image, f 1 is the spatial size of a filter. B 1 is an n 1 -dimensional vector, each element of which is associated with a filter. Intuitively, W 1 applies n 1 convolutions on the image, then combines with the n 1 -dimensional bias vector B 1 to increase degrees of freedom by 1. Finally, the output is composed of n 1 feature maps and will be passed to the second step.

Non-linear mapping The purpose of this step is to increase the resolution of each patch to obtain a high-resolution patch. In the second operation, we map each of the n 1 -dimensional vectors from the first layer to a n 2 -dimensional one. The operation of the second layer is as follows:

F 2 (X L ) = max(0, W 2 * F 1 (X L ) + B2). (6.1.2) W 2 contains n 2 filters of size n 1 × f 2 × f 2 , and B 2 is n 2 -dimensional bias vector.
Each n 2 -dimensional output vector is a representation of a HR patch that will be used for the reconstruction step. change challenge Reconstruction The reconstruction step aggregates all HR patches to restore the HR image by using the following convolutional layer Generator The generator (G) learns to generate plausible data. The input of this network is a random sample, and the output is a generated sample which will be used in the discriminator network. The objective of the generator is to fool the discriminator network into thinking that the generated sample is real.

F 3 (X L ) = max(0, W 3 * F 2 (X L ) + B3). ( 6
L(Θ) = 1 n n i=1 ∥F 3 (X L,i ) -X H,i ∥ 2 , ( 6
Discriminator The discriminator (D) learns to distinguish the fake data of the generator from the real data. This network is a basic classification network that returns the probability that the input sample belongs to the database (that it is real). Ideally, we consider a sample y. D(y) = 1 if y belongs to the database and D(y) = 0 if y is a generated sample.

?figurename? 6.4: Sketch of the GAN architecture [START_REF] Goodfellow | Generative adversarial nets[END_REF].

Loss function To define the generator G and discriminator D, we assume that x is ground truth data and that z is observed data. The objective of D is to properly classify the real and generated data, this amounts to maximizing equation

max D log(D(x)) + log(1 -D(G(z))). (6.2.1)
At the same time, the objective of G is to deceive the discriminator, that is, G is trained to minimize the probability that the discriminator makes the correct prediction for a generated data set, so we want D(G(z)) = 1. This amounts to having to minimize log(1-D(G(z))) with respect to the parameters of G. Therefore, D and G play the following two-player minimax game with the value function V (G, D):

min G max D V (G, D) = min G max D E x [log D(x)] + E z [log(1 -D(G(z)))]. (6.2.2)
The optimization of the weights of the generator θ G and the discriminator θ D is carried out in an alternative way, optimizing in D and then in G from equation (6.2.2) as in Algorithm 4 [START_REF] Goodfellow | Generative adversarial nets[END_REF]. In this algorithm, the parameters are updated using the stochastic gradient descent method with their stochastic gradient:

∇ θ G 1 m m i=1 log(1 -D(G(z (i) ))) = 1 m m i=1 - 1 1 -D(G(z (i) )) D ′ (G(z (i) ))∇G(z (i) ) , (6.2.3) ∇ θ D 1 m m i=1 log D(x (i) ) + log(1 -D(G(z (i) ))) = 1 m m i=1 1 D(x (i) ) ∇D(x (i) ) - 1 1 -D(G(z (i) ))
∇D(G(z (i) )) , (6.2.4)

92

?chaptername? 6. A generative adversarial network with contrast change challenge where {z (1) , . . . , z (m) } are the observed samples and {x (1) , . . . , x (m) } are real examples.

Algorithm 4 Minibatch stochastic gradient descent training of generative adversarial nets [START_REF] Goodfellow | Generative adversarial nets[END_REF] Input: noise samples {z (1) , . . . , z (m) } and examples {x (1) , . . . , x (m) } for number of training iterations do for k steps do

• Sample minibatch of m noise samples {z (1) , . . . , z (m) } from noise prior p g (z).

• Sample minibatch of m examples {x (1) , . . . , x (m) } from data generating distribution p data(x) .

• Update the discriminator by ascending its stochastic gradient:

∇ θ D 1 m m i=1 log D(x (i) ) + log(1 -D(G(z (i) ))) .
end for

• Sample minibatch of m noise samples {z (1) , . . . , z (m) } from noise prior p g (z).

• Update the generator by descending its stochastic gradient:

∇ θ G 1 m m i=1 log(1 -D(G(z (i) ))).

end for

In practice, the hyperparameter k is chosen equal to 1 and the Stochastic Gradient Descent SGD step is done using the ADAptive Moment estimation (ADAM) algorithm [START_REF] Kingma | ADAM : A Method for Stochastic Optimization[END_REF]. ADAM algorithm is an adaptive stochastic gradient descent optimization algorithm specifically designed for training deep neural networks. It is an algorithm in which the gradient used in each iteration is updated from the previous one using a moment-based technique.

SRGAN method [56]

For the super-resolution task, Ledig et al. [START_REF] Ledig | Photo-realistic single image superresolution using a generative adversarial network[END_REF] pioneered a super-resolution method using GAN. This method is called SRGAN. The core concept of GAN is retained in SRGAN, i.e. the generator G generates a estimated HR image (conditionally for certain LR images) and the discriminant operator classifies it as a ground truth HR image or fake HR image.

SRGAN Architecture

In order to obtain generator G and the discriminator D, the authors use a GAN architecture (as shown in Figure 6.5) for the generator and a classification network combining dense and convolutional layers for the discriminator.

?figurename? 6.5: Architecture of the GAN [START_REF] Ledig | Photo-realistic single image superresolution using a generative adversarial network[END_REF] with the corresponding kernel size k, the number of feature maps n and the stride s indicated for each convolution layer.

Generator Architecture The generator architecture contains B residual blocks with 16 identical layouts, created by a residual neural network (ResNet). Specifically, within the residual block, two convolutional layers are used, with small 3×3 kernels and 64 feature maps followed by batch-normalization (BN) layers [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] and ParametricReLU (PReLU) [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF] as the activation function. The PReLU adaptively learns rectifier parameters and improves accuracy with negligible computational cost. Then, the LR image is up-resolution using two trained sub-pixel convolutional layers [START_REF] Shi | Real-time single image and video superresolution using an efficient sub-pixel convolutional neural network[END_REF]. Finally, the output of the generator architecture is passed into the discriminator.

Discriminator Architecture The task of the discriminator is to discriminate between real HR images and generated SR images. This architecture contains 8 convolutional layers with an increasing number of 3 × 3 filter kernels, increasing by a factor of 2 from 64 to 512 kernels. In this network, the LeakyReLU activation function is used. To reduce the resolution of the image for distinguishing step, we use the strided convolutions each time the number of features is doubled. Then two dense layers with a ReLU function are used in between. Finally, a sigmoid activation function is added at the end of the network to obtain a probability for sample classification.

Loss function

The SRGAN uses the perpetual loss function l SR for the generator network. The perceptual term is motivated by the fact that a perceptual loss function gives visually ?chaptername? 6. A generative adversarial network with contrast change challenge better results than a loss function in pixel space. The authors then propose the loss function, which is the weighted sum of two loss components: content loss and adversarial loss. l SR = l SR X + 10 -3 l SR Gen , (6.2.5)

where l SR X is a content loss and l SR Gen is a adversarial loss.

Content loss Let the LR image X L that serves as input to the generator network G and the reference image X H , the content loss function is a VGG loss based on the ReLU activation function of the pre-trained 19-layer VGG network [START_REF] Simonyan | Very deep convolutional networks for largescale image recognition[END_REF].

l SR V GG/i,j = 1 W i,j H i,j W i,j m=1 H i,j n=1 [ϕ i,j (X H ) m,n -ϕ i,j (G(X L )) m,n ] 2 (6.2.6)
W i,j and H i,j correspond to the size of the image at the output of the layer ϕ i,j .

Adversarial loss The adversarial loss is the loss function that forces the generator to an image more similar to HR image by using the trained discriminator network to differentiate realistic images from unrealistic ones. This loss function is expressed as

l SR Gen = N n=1 -log(D(G(X L ))) (6.2.7)
where N corresponds to the batch size.

Contribution: Residual dense GAN (RDGAN) network for contrast change problem of superresolution

The problem of changing contrast is one of the important challenges of SR for materials images. In the case where the transformation between LR and HR image is an affine transformation, we can estimate the contrast change parameters of this transformation as described in Chapter 3. Moreover, in Chapter 4 we proposed an SR method, the MMSE-GGMM method, which is invariant under affine transformations of the observation. However, we are not sure that the change of contrast is always characterized by a simple affine transformation. It would be interesting to show some examples of non-affine transformations. To deal with the non-affine transformation, we propose an SR method based on GAN considering a bi-generator network that includes two generator sub-networks. The first generator deals with the contrast change problem, and the second is to reconstruct the high frequencies. The architecture can be summarized as in Figure 6.6: In the SR training step for material images, we have a data set of HR and LR pair images, which have different contrast and brightness. The first generator of our GAN network aims to transfer the contrast of the LR image X L to a contrast as good as the HR image X H . It means, it will be necessary to associate each pixel of LR image with the pixels of the HR image of which it is closest in terms of distance.

To do this, we will use optimal transport for the LR image and downsampling of the HR image so that its resolution is equal to the resolution of the LR image. We denote the downsampling of the HR image as ↓ X H . The way to achieve the optimal overall cost is to change the range of the LR image to be the same as the HR image. Therefore, we minimize the distance between the sorted value in pixels of two images X L and ↓ X H . The first loss function corresponding to G 1 is:

L(G 1 ) = N n=1 ||sort(↓ X H ) -sort( XL )|| 1 (6.3.1) 6.3.1.2 Second generator G 2
The second generator sub-network takes care of the high frequency reconstruction and returns the final HR image XH . As the goal of SR for materials images is to understand textures and interfaces in materials, we consider that the SR approach focuses on the image geometry. To enhance the geometric reconstruction of the images, we propose to consider a regularization term in the generator loss function. This term was initially proposed by Ballester et al. [START_REF] Ballester | A Variational Model for P+XS Image Fusion[END_REF] in a variational framework.

We then use the loss function, which is proposed by Gastineau et al. [START_REF] Gastineau | A Residual Dense Generative Adversarial Network for Pansharpening with Geometrical Constraints[END_REF], for this subnetwork with three terms of the loss function such as adversarial loss, content loss, and geometric loss. The loss function for the second sub-network G 2 is expressed as ?chaptername? 6. A generative adversarial network with contrast change challenge follows:

L(G 2 ) = N n=1 -log(D( XH )) + α l1 || XH -X H || 1 + α g x∈Ω |⟨∇X H (x) ⊥ , ∇ XH (x)⟩| (6.3.
2) where N corresponds to the batch size, ∇(•) is the gradient, ⊥ the orthogonal vector, Ω the image domain, and α L 1 , α g are the content weight and the geometric weight respectively. The first term is the adversarial loss that is associated with the discriminator. The second term is the l 1 norm between the target and the reconstructed image. This third term forces the alignment of the gradient of the reconstructed image with the gradient of the target image. This allows the geometry of the reference image to be transferred to that of the one we are trying to reconstruct. Then the scalar product is used for the orthogonal gradient vector of the target image and the gradient vector of the reconstructed image at any point in the image domain. Indeed, a zero scalar product indicates that the vectors are collinear and therefore the direction of the gradients is preserved. The architecture starts with a convolutional layer with 32 3 × 3 filter kernels, 1 stride, and ReLU activation function. This is followed by p residual blocks RB i as in Figure 6.8. In each residual block, we use 3 convolutional layers with an increasing number of 3 × 3 filter kernels from 32 to 128, stride 1, and ReLU as the activation function. This block is terminated by a convolutional layer with 32 1×1 filter kernel, stride 1. The idea of the residual network is that each layer of the neural network is only responsible for fine-tuning the output of a previous layer by simply adding the residual learned at the previous layer to the input of that layer. Finally, the architecture of our first generator is completed with a convolution layer with one 1 × 1 filter kernel and stride 1.

Second generator We consider the same architecture as in the RDGAN method [START_REF] Gastineau | A Residual Dense Generative Adversarial Network for Pansharpening with Geometrical Constraints[END_REF], a residual dense architecture as in Figure 6.9.

?figurename? 6.9: Architecture of the second generator, where blocks B i , i = 1, • • • , p, are p residual dense blocks, k is the kernel size, n the number of filters and s the stride of the convolutional layers. Arrows represent the dense connections and the ⊕ residual connection.

The architecture starts with a convolutional layer with 32 3 × 3 filter kernels and a ReLU activation function. This is followed by p residual dense blocks B i , i = 1, • • • p, which retain the advantages of Densenet [START_REF] Huang | Weinberer Densely connected convolutional networks[END_REF] and residual [START_REF] He | Deep residual learning for image recognition[END_REF] architectures. Thus, this architecture can avoid the problem of vanishing gradients that is often encountered during the training step. Each residual dense block is composed of four convolutional layers. In each block, the input to the l th layer consists of the feature maps of all previous layers. It means that if we note H l the non-linear transformation given by layer l of the network and x l the output of this layer, the l th layer receives the feature maps of all the previous layers, x 0 , x -1, • • • , x l-1 , as input:

x l = H l ([x 0 , x 1 , • • • , x l-1 ]) (6. In addition, each of these blocks consists of a residual connection. This connection is represented by adding the input of each block with the output of the last layer. Therefore, the input information for each block is transmitted without modification.

After extracting the features in the LR space, we up-sample these features into the HR space by using the sub-pixel convolutional layer as proposed by Shi et al. [START_REF] Shi | Real-time single image and video superresolution using an efficient sub-pixel convolutional neural network[END_REF]. Finally, the architecture of our second generator is completed with one convolution layer with one 1 × 1 filter kernel and stride 1 as in the first generator.

Discriminator

Loss function For the discriminator, we maximize the loss function as follows:

L(D) = N n=1
log(1 -D(G(X L ))) + log(D(X H )).

(6.3.4)

Architecture of discriminator We consider the same architecture of the discriminator network as in the MDSSC-GAN SAM method [START_REF] Gastineau | Generative adversarial network for pansharpening with spectral and spatial discriminators[END_REF]. The first convolutional layer of the architecture performs a standard convolution with 32 feature maps, 1 stride, and a ReLU activation function. This is followed by an architecture containing 6 convolutional layers with an increasing number of 3 × 3 filter kernels, increasing by a factor of 2 from 32 to 1024 kernels. Each layer is followed by the BN layer and the ReLU activation function with stride 2, except the last layer with 1 stride. These convolutional layers are used to extract sufficient features and capture the representation of data in a space. Then, two dense layers ?figurename? 6.11: Architecture of the discriminator, where k is the kernel size, n the number of filters and s the stride of the convolutional layers.

Numerical results

Database In this contribution, we implement the RDGAN-contrast method for our material images: The metric of total variation allows us to compare the edges of two images.

Details of implementation

The proposed method is implemented with Tensorflow 1.2 and uses the ADAM algorithm to optimize the weights of the networks with an initial learning rate of 0.0002 and a momentum of 0.5. Finally, the batch size is adjusted to 10 for our database. Parameters α g , α l1 are optimized to get the best balance between all the metrics.

Results

We compare the proposed method with some state-of-the-art SR methods. The SR methods in comparison are the bicubic interpolate method, MMSE-GGMM method, RDGAN and our proposal RDGAN-contrast method. For a better comparison, we train both networks on the FS and Sic Diamonds images, which have a contrast change between the LR and HR images. We know that the MMSE-GGMM and RDGAN-contrast methods can directly solve the contrast change problem in the case of an affine transformation and a nonaffine transformation, respectively. But the bicubic and RDGAN methods cannot. Therefore, to make the comparison fair for the bicubic and RDGAN methods, we first estimate the contrast change parameters using the estimation method as in Section 3.2.2 for all pairs of HR and LR images from the training data set. Thus, we apply the affine transformation within these parameters to the reconstructed HR images, which were obtained by the bicubic and RDGAN method. Table 6.1 shows the PSNR and the metric total variation values of the reconstruction of the FS and Sic Diamonds images using either bicubic interpolation, MMSE-GGMM, RDGAN, and RDGAN-contrast with magnification factor q = 2.

FS

For the bicubic and RDGAN methods, we first estimate the contrast parameters from the training data. Then we apply the affine transformation to the recontructed HR images. In the FS image, the PSNR value of the RDGAN method is slightly higher than our proposal RDGAN-contrast. Its TV value is lower than that of RDGAN-contrast. However, both the PSNR and TV values of our RDGAN-contrast method are superior to the other methods for the Sic Diamond image. This means that when the contrast transformation is affine, the RDGAN-contrast method can obtain results similar to those of the MMSE-GGMM and RDGAN methods. But when the contrast transformation is a non-affine, the RDGAN-contrast is more efficient than the other methods both in contrast and geometry. This can be seen in more detail in Figure 6.12. In addition, in the case of FS image, our MMSE-GGMM method results in only 0.1 dB lower than the RDGAN method. However, for model learning, the MMSE-GGMM method only needs to extract the database from a 2D image. As for the RDGAN and RDGAN-contrast methods, they need to use the overlapping patches from 40 2D images.

Conclusion

In this chapter, we have studied the deep learning approach for super-resolution. We present a new SR method based on a residual dense generative adversarial neural network (RDGAN) with the contrast change problem, called RDGAN-contrast.

To solve the contrast change problem, our method considers two generator subnetworks. The first sub-network focuses on the contrast change problem and the second one reconstructs the high frequencies. The results show a visual improvement with our proposal RDGAN-contrast over the RDGAN method and the MMSE-GGMM method in the case of non-affine transformation. However, in the case of the affine transformation, the MMSE-GGMM method should be preferred to the RDGAN-contrast method. change challenge ?chaptername? 7

Conclusions and perspectives

Conclusions

In this thesis, we are interested in solving the super-resolution problem for material images using learning methods.The three main challenges of the SR problem for material images are the contrast change problem, the problem of undefined corruption operators, and the high dimensionality of the data. We have proposed four contributions based on the MMSE estimator for GGMM, the PCA method for GMM and GGMM, and the GAN method to solve these challenges. The first proposed method [START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF] mainly addresses our first dual challenge of material images by considering the MMSE estimator with the GGMM model. The results obtained for the FS and Sic Diamonds images have shown that the MMSE-GGMM method is invariant with the contrast change problem under affine transformations of the observation. We also proved that our SR method using the MMSE estimator with the GGMM model improves the results over the GMM model in both synthetic and material images.

The second contribution introduced a method to deal with the last challenge of our material images. This is the problem of high dimensionality. Our contribution [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] combined the GMM model with a reduction in the dimensionality of the data in each component of the model using the principal component analysis method, called PCA-GMM. Next, this PCA-GMM model is applied to the super-resolution of 2D and 3D material images based on the MMSE-GMM method. The numerical results of this second contribution confirm the gain of the PCA-GGMM method in reducing the dimension of the patches when we learn the mixture model. In addition, they maintain almost the same quality of results as with a GMM algorithm for the super-resolution problem.

Following these second results, our third contribution [START_REF] Nguyen | Image superresolution with PCA Reduced generalized Gaussian mixture models[END_REF] is an extension of the second one by focusing on the GGMM model with the PCA method. Due to the complexity of the GGMM model, it is not easy to consider a GGMM in conjunction ?chaptername? 7. Conclusions and perspectives with PCA in the same way as in the second contribution. This PCA-GGMM takes advantage of the flexibility of the "Half-Quadratic splitting" technique to tackle the minimization problem in the M-step of the EM algorithm. Then the PCA-GGMM model can be used for the super-resolution model based on the MMSE method for GGMM as in the first contribution. The PCA-GGMM model produces results almost as good as the MMSE-GGMM model and even slightly better than the MMSE-GMM model.

Finally, the last proposed method mainly focuses on SR method for material images with the contrast change problem based on the deep learning approach, specifically the SRGAN method. Indeed, this contribution considers two generators, each responding to one of the two problems: contrast change and the superresolution task. The results of the RDGAN-contrast method obtained on the material images are encouraging because we can observe a quantitative gain confirmed by the visual results in both contrast and texture. However, in the case of the affine transformation, the RDGAN-contrast method requires a larger database and is more computationally complex than the MMSE-GGMM method, but it is slightly better than the MMSE-GGMM method. Thus, if the contrast change is a non-affine transformation or if we have a large database, we prefer to use the RDGAN-contrast method. Conversely, if the contrast change is an affine transformation or if we have little data, the MMSE-GGMM method should be preferred.

Perspectives

This thesis provides some proposals for the super-resolution problem with several challenges to material images. However, some other possibilities remain unexplored such as the SR method with the elliptical mixture model, the super-resolution problem with a high magnification factor, and the convergence of the GANs.

A first perspective could be to consider a dimensionality reduction method within mixture models of elliptical distributions. Indeed, throughout this thesis, we have been interested in dimensionality reduction methods for the GMM and GGMM models. The generalized Gaussian distribution is a special case of the elliptical distribution. Thus, the PCA-GGMM model can be generalized to the PCA-reduced elliptical mixture model by combining PCA with the elliptical mixture model. Then the PCA-reduced elliptical mixture model can be used for super-resolution task by using the MMSE estimator following the same methodology as the MMSE-GGMM method.

The second perspective would be to study the convergence of the GANs. In the last contribution, we have proposed an SR method based on GANs for the contrast change problem. On the other hand, GANs are rather unstable algorithms and may have difficulty in converging. The reason for these problems is that one of the networks can overtake the adversarial network. In this case, the less efficient network will not be able to progress. Thus, the training will become unstable. In

List of publications 105

this thesis, we try to solve this issue by using the residual dense network for the second generator. Recently, a new method [START_REF] Zadorozhnyy | Adaptive weighted discriminator for training generative adversarial networks[END_REF] using adaptive weighting for the discriminator has been developed to solve the convergence problem. The author has shown that this method successfully remedies the problem of instability and convergence of GANs. Indeed, our RDGAN-contrast network can be improved by using the discriminator loss, which consists of two different weighted parts: the real and the fake terms.

From the material imaging point of view, the magnification factor can be up to 5, 10. As the scale factor increases, some essential details may be lost, which has a serious impact on the material image. The large scale factor is also one of the major challenge in the SR community and there is still a lack of effective remedies. Recently, some methods for super-resolution with large scale factor were proposed such as StyleGAN [START_REF] Karras | A stylebased generator architecture for generative adversarial networks[END_REF], PULSE [START_REF] Menon | PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models[END_REF] method, and GLEAN [START_REF] Chan | Generative latent bank for large-factor image super-resolution[END_REF] method. These methods significantly improve the restoration quality of large-factor image superresolution by using the latent space. These methods have achieved the improvements in fidelity and more realistic textures than existing methods for SR with a large scale factor. These contributions open up a very promising direction for image SR with a high scale factor, especially for material images.

List of publications

  -Thanh NGUYEN, Dr. Duc-Thach-Son VU, Mr. Viet-Anh NGUYEN, Mrs. Minh-Thi NGUYEN, Miss Huyen-Mai PHAM, Dr. Thuy-Ha PHAM, Mrs. Nhat-Vy NGO, Miss Thanh-Xuan NGUYEN, Mr. Huu-Nhut LE, Dr. Phuong-Yen TRAN, Mrs Simone Dang Van and her husband.

3 . 4

 34 Cross sections of Fontainebleau sandstone with different resolutions. a) 2x MAG with pixel size of 3.25µm, b) 4x MAG with pixel size of 1 : 63µm, c) and d) 10x MAG with pixel size of 0.65µm . . . . . . . 19 3.5 Top: Material "FS", bottom: Material "SiC Diamonds". First column: Ground-truth HR image, second column: Input LR image, third column: Match LR images. The red lines in the second column are the boundaries of the LR image that correspond to the HR image. 20 3.6 Matched points between the HR and LR image by using SIFT . . . 21 3.7 Histogram of the images of the FS material. The image on the left is the histogram of the high-resolution image, and the image on the right is its low-resolution correspondence. . . . . . . . . . . . . . . . 21 3.8 Sketch of the overall framework of SR. . . . . . . . . . . . . . . . . . 22 4.1 Top: Images for estimating the mixture models. Bottom: Ground truth for reconstruction. First column: Material "FS", second column: Material "SiC Diamonds", third column: goldhill image. . 42 4.2 Reconstructions of 2D Sic Diamonds image with magnification factor q = 2 by using MMSE and EPLL method. . . . . . . . . . . . . . . . 45 5.1 Plot of ordered eigenvalues of S k (left) and differences between eigenvalues (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 viii ?listfigurename?

80 6. 1

 801 First row: ground truth, second row: low resolution, third row: reconstruction with GMM, fourth row: reconstruction with PCA-GMM and d = 20, fifth row: reconstruction with PCA-GMM and d = 12. The larger of d, the closer is the result of PCA-GMM to GMM. . . . . . . . . . . . 79 5.3 Histograms of the eigenvalues of Σ k , k = 1, ..., K for the PCA-GMM with fixed σ = 0.02 for d = 20. a)FS with magnification q = 4, b) SiC Diamonds with magnification q = 4 . . . . . . . . . . . . . . . . Scheme of max pooling . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.2 Scheme of average pooling . . . . . . . . . . . . . . . . . . . . . . . . 88 6.3 Sketch of the SRCNN architecture [24]. . . . . . . . . . . . . . . . . . 89 6.4 Sketch of the GAN architecture [35]. . . . . . . . . . . . . . . . . . . 91 6.5 Architecture of the GAN [56] with the corresponding kernel size k, the number of feature maps n and the stride s indicated for each convolution layer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.6 Scheme of the proposed architecture, where X H is the HR image, X L the LR image, XL the LR image with the good contrast and XH the SR image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.7 Architecture of the the first generator, where blocks RB i , i = 1, • • • , p, are p residual blocks, k is the kernel size, n the number of filters and s the stride of the convolutional layers. . . . . . . . . . 96 6.8 Architecture of each residual block RB i , i = 1, • • • p. . . . . . . . . . . 97 6.9 Architecture of the second generator, where blocks B i , i = 1, • • • , p, are p residual dense blocks, k is the kernel size, n the number of filters and s the stride of the convolutional layers. Arrows represent the dense connections and the ⊕ residual connection. . . . . . . . . . 97 6.10 Architecture of the Dense network. A 5-layer dense block with a growth rate of k = 4. Each layer takes all preceding feature-maps as input. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 6.11 Architecture of the discriminator, where k is the kernel size, n the number of filters and s the stride of the convolutional layers. . . . . . 99 6.12 Reconstructions of 2D low-resolution images with a magnification factor q = 2. First row: LR image, second row: ground truth, third row: reconstruction with Bicubic, fourth row: reconstruction with MMSE-GGMM, fifth row: reconstruction with RDGAN, sixth row reconstruction with RDGAN-contrast. . . . . . . . . . . . . . . . . . 102

1. 2 .

 2 Motivations et objectifs 3 ?figurename? 1.1: Rendu 3D d'une électrode de pile à combustible avec, sur la partie gauche, la porosité (transparente), le conducteur électronique (vert) et le conducteur ionique (jaune). Sur la partie droite, les lignes rouges représentent les Triple-Phase-Boundaries où les réactions électrochimiques ont lieu.

Figure 3 . 1 .

 31 In this example, we keep the level of contrast and blur of the image constant, and reduce the resolution from 272 × 272 to 34 × 34. The three last images are generated using the downsampling operator as shown in equation (3.2.3) with magnification factors q = 2, 4, 8, respectively. ?figurename? 3.1: Cameraman image with different resolution from the left to the right: 272 × 272, 136 × 136, 68 × 68, and 34 × 34 (keep noise and contrast level constant). Image noise Image noise is a random variation of brightness or color information in an image. It is caused by the image sensor and the circuits of a camera. Figure 3.2 shows the effect of noise when we change the level of noise. ?figurename? 3.2: Cameraman image size 272 × 272 with different level of noise (keep resolution and contrast level constant). Contrast image Contrast is the difference in luminance or colour that represents in an image. Good contrast means that the grey values present in the image range from black to white, making use of the full range of brightness to which the human vision system is sensitive. The different contrasts of the image are illustrated in Figure 3.3. ?figurename? 3.3: Cameraman image with different contrasts (keep resolution constant and no noise).Spatial Super-resolution Super-resolution (SR) is a class of image restoration that increases the resolution of image. According to the number of input LR images, SR can be classified into single-image super-resolution (SISR) and multiimage super-resolution (MISR). In the context of this thesis, we refer to SISR, which allows to increase the spatial resolution of the image from a single LR image.

3 . 4 :

 34 Cross sections of Fontainebleau sandstone with different resolutions. a) 2x MAG with pixel size of 3.25µm, b) 4x MAG with pixel size of 1 : 63µm, c) and d) 10x MAG with pixel size of 0.65µm

Figure 3 .

 3 5 clearly show this comment. In addition to that, Figure 3.7 shows the histogram of the HR and LR images of FS material data. It shows that the contrasts of the HR and LR images are different. ?figurename? 3.7: Histogram of the images of the FS material. The image on the left is the histogram of the high-resolution image, and the image on the right is its low-resolution correspondence.

4. 2 . 3 . 1

 231 Dataset and quality evaluationGeneration of the dataset. In the experiment, for a given high-resolution image X H , we now generate the low-resolution image X L by X L = AX H +ε, where epsilon is a realization of the white Gaussian noise with standard deviation σ = 0.02. For the downsampling operator A, we use the definition in Equation (3.2.2). The blur operator H is given by a convolution with a Gaussian kernel with standard deviation 0.5.?figurename? 4.1: Top: Images for estimating the mixture models. Bottom: Ground truth for reconstruction. First column: Material "FS", second column: Material "SiC Diamonds", third column: goldhill image. Generation of the training dataset. To estimate the parameters of the mixture model, we used a part of the original HR image, such as the top left quarter as the top row in Figure 4.1 and the corresponding part of the LR image. Slices of the 3D images and the basic image for training step are given Figure 4.1. From the training part of the LR image X L , we define a set of overlapping patches of size τ

4 . 2 :

 42 Reconstructions of 2D Sic Diamonds image with magnification factor q = 2 by using MMSE and EPLL method.

2 . 1 )

 21 for b ∈ R n and U ∈ St(d, n), where St(d, n) := {U ∈ R n,d : U T U = I} is the Stiefel manifold. By setting the derivative of P with respect to b equal to zero, the offset (bias) b = x := 1 N (x 1 + . . . + x N ). So that we can reduce our attention to the minimization with respect to U ∈ St(d, n), i.e., to the consideration of

. 9 )

 9 where p (x | k) is a single PPCA model (5.2.4) and w k is the corresponding cluster weight, with w k are non-negative and sum up to one for all k cluster. As a consequence, the parameters U k , b k , σ 2 k are now associated with each of the K mixture components. This means that the generative model (5.2.2) now requires the weight w k , and the appropriate parameters U K , b k , σ 2 k of each mixture component. These parameters can be estimated by using an iterative EM Algorithm 1 for the mixture model to maximize the log-likelihood function (5.2.9) of all of the model parameters w k , U k , b k , σ 2 k . The parameters θ k in equation (4.2.4) now are considered as θ k = U k , b k , σ 2 k . At iteration r, the posterior responsibility of mixture k of x in the E-Step is given by α

  .2.10) and the mixture component weight w k is estimated as in equation (4.2.3). The parameter b k is updated as b

  r+1) k T . • Sub-space E k : the d k first columns of Q k are estimated by the eigenvectors associated to the d k largest eigenvalues λ kj of the empirical covariance matrix S (r+1) k . • The parameters a kj are estimated by the d k largest eigenvalues λ kj of S (r+1) k . • The parameter of b k is estimated by: b

3 . 35 )Corollary 1 .

 3351 with θk = (μ k , Σk ) as in(5.3.21).(5.3.36) end for For the iterates U (r) , b (r) , w (r) , θ (r) r generated by Algorithm 2 the objective function F is decreasing. The interesting step is the second M-Step which requires again the maximization of a function. Based on (4.2.6) and (4.2.7) we can prove the following proposition.

3 . 37 )

 337 with θ = (μ, Σ) of the form (5.3.8) and (5.3.9) is given by μ = 0, Σ = 1 w Û T S Û , and b = 1 w m, (5.3.38) 5.3. Contribution: Dimensionality-Reduced Gaussian Distribution: PCA-GMM Model [41] 61

2 1 w

 21 tr(U T SU ) + w log(|U T SU |). (5.3.41) ii) If σ is learned, we have σ2 = (n-d) tr(S) -tr( Û T S Û ) , (5.3.42) and G from (5.3.41) is replaced by G(U ) := (n -d) log tr(S) -tr(U T SU ) + log(|U T SU |). (5.3.43)

5. 4 .

 4 Contribution: Dimensionality-Reduced Generalized Gaussian Distribution: PCA-GGMM Model[START_REF] Nguyen | Image superresolution with PCA Reduced generalized Gaussian mixture models[END_REF] 71

  (r+1) ) , where L1 (b (r) ) and L2 (U (r+1) ) are estimates of the Lipschitz constant of ∇ U G(•, b(r) ) and ∇ b G(U (r+1) , •).

  First row: ground truth, second row: low resolution, third row: reconstruction with GMM, fourth row: reconstruction with PCA-GMM and d = 20, fifth row: reconstruction with PCA-GMM and d = 12. The larger of d, the closer is the result of PCA-GMM to GMM.

  ?figurename? 5.3: Histograms of the eigenvalues of Σ k , k = 1, ..., K for the PCA-GMM with fixed σ = 0.02 for d = 20. a)FS with magnification q = 4, b) SiC Diamonds with magnification q = 4

.1. 3 )W 3

 33 and B 3 here correspond to c filters of size n 2 × f 3 × f 3 and c-dimensional bias vector, respectively. This layer therefore reconstructs the HR image by analogy with the weighted average for high-resolution patches.Although the three operations of the SRCNN architecture are motivated by different objectives, they all lead to the same form of convolutional layer with a ReLU activation function. In this model, all filter weights and biases Θ = {W 1 , W 2 , W 3 , B 1 , B 2 , B 3 } are learned by minimizing an objective function related to the mean squared error (MSE) between the HR training image and the corresponding reconstructed HR image:

6. 3 .

 3 Contribution: Residual dense GAN (RDGAN) network for contrast change problem of super-resolution 95 ?figurename? 6.6: Scheme of the proposed architecture, where X H is the HR image, X L the LR image, XL the LR image with the good contrast and XH the SR image. 6.3.1 Generator 6.3.1.1 First generator G 1

6. 3 . 1 . 3

 313 Architecture of generator First generator The architecture of the first generator, presented in Figure 6.7, is composed of two convolutional layers and RB i , i = 1, • • • , p residual blocks.?figurename? 6.7: Architecture of the the first generator, where blocks RB i , i = 1, • • • , p, are p residual blocks, k is the kernel size, n the number of filters and s the stride of the convolutional layers.

6. 3 .

 3 Contribution: Residual dense GAN (RDGAN) network for contrast change problem of super-resolution 97 ?figurename? 6.8: Architecture of each residual block RB i , i = 1, • • • p.

3 . 3 )

 33 where [•] represents the concatenation, and [x 0 , x -1, • • • , x l-1 ] refers to the concatenation of the feature-maps produced in layers 0, • • • , l -1. change challenge ?figurename? 6.10: Architecture of the Dense network. A 5-layer dense block with a growth rate of k = 4. Each layer takes all preceding feature-maps as input.

6. 3 .

 3 Contribution: Residual dense GAN (RDGAN) network for contrast change problem of super-resolution 99 are used to classify whether the generated image is real or fake as in the SRGAN architecture.

  FS and Sic diamonds. Due to the complexity of the network, we cannot train the network from overlapping patches of size 4 × 4 as in the MMSE-GGMM experiment. Therefore, we train the network with a patch size of 64 × 64 for LR patches. This leads to the use of the training database from a 2D image for training is not sufficient. Fortunately, our 3D image was acquired with several slices of the 2D image. For instance, we obtain a 3D HR image which has a size of 1652 × 1652 × 40. This gives us 40 2D HR images of size 1652 × 1652. Then, for each 2D HR image, we have a corresponding 2D LR image of size 826 × 826. To generate the database for the training step, we use the upper left quarter of 40 pairs of HR and LR images as in the top row of Figure 4.1. Finally, we extract the quarter of these HR images into a set of HR patches with patch size 128 × 128 without overlapping and LR patches with size 64 × 64. Therefore, we can obtain a training set with 1960 pairs of HR and LR images.For the test process, we reconstruct the HR image for the entire image with a pixel size of 1652 × 1652 from one of the 40 2D images that we obtained from the original 3D image. We again extract this image and the corresponding LR image into a set of HR patches of size 128 × 128 and LR patches of size 64 × 64 to feed to the generator network. Thus, we will apply our network to these 225 pairs of HR LR images for testing.Quality evaluation To evaluate the results, we use the Peak Signal to Noise Ratio (PSNR) value in equation (4.2.27) of two images X and Y . In addition, in order to compare the geometric similarities of two images X and Y , we consider the ?chaptername? 6. A generative adversarial network with contrast change challenge metric of total variation. The total variation (TV) is given asT V (X, Y ) = 1 |Ω| Ω ||∇X -∇Y || 2 = 1 |Ω| Ω |∇ x X -∇ x Y | 2 + |∇ y X -∇ y Y | 2 , (6.3.5) where |Ω| is the number of pixels. The l 2 norm ||X -

  Diamonds part?figurename? 6.12: Reconstructions of 2D low-resolution images with a magnification factor q = 2. First row: LR image, second row: ground truth, third row: reconstruction with Bicubic, fourth row: reconstruction with MMSE-GGMM, fifth row: reconstruction with RDGAN, sixth row reconstruction with RDGANcontrast.

Contents 7 . 1

 71 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 7.2 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 7.3 List of publications . . . . . . . . . . . . . . . . . . . . . . . . 105

  Hertrich, D-P-L. Nguyen, J-F. Aujol, D. Bernard, Y. Berthoumieu, A. Saadaldin, and G. Steidl. PCA Reduced Gaussian Mixture Models with Applications in Superresolution. Inverse Problems in Imaging, 16 (2) : 341-366. doi: 10.3934/ipi.2021053, 2022. Submitted journal • D-P-L. Nguyen, J. Hertrich, J-F. Aujol and Y. Berthoumieu. Image superresolution with PCA Reduced generalized Gaussian mixture models. HAL Preprint hal-03664839, 2022. Conference paper • D-P-L. Nguyen, J-F. Aujol and Y. Berthoumieu. Image Super Resolution with a Joint Generalized Gaussian Mixture Model. XXVIIIe Colloque Francophone de Traitement du Signal et des Images (GRETSI), September, 2022.

  1.1 Rendu 3D d'une électrode de pile à combustible avec, sur la partie gauche, la porosité (transparente), le conducteur électronique (vert) et le conducteur ionique (jaune). Sur la partie droite, les lignes rouges représentent les Triple-Phase-Boundaries où les réactions électrochimiques ont lieu. . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1 3D rendering of a fuel cell electrode with, on the left part, porosity (transparent), electronic conductor (green) and ionic conductor (yellow). On the right part the red lines represent the Triple-Phase-Boundaries where electro-chemical reactions take place. . . . . . . . . 11

3.1 Cameraman image with different resolution from the left to the right: 272 × 272, 136 × 136, 68 × 68, and 34 × 34 (keep noise and contrast level constant). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.2 Cameraman image size 272 × 272 with different level of noise (keep resolution and contrast level constant). . . . . . . . . . . . . . . . . . 17 3.3 Cameraman image with different contrasts (keep resolution constant and no noise). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

  Contents 1.1 Contexte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Motivations et objectifs . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 Motivation en sciences des matériaux . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . 83 6.1 Quantitative results obtained on FS and Sic Diamonds images, the best results are in bold. . . . . . . . . . . . . . . . . . . . . . . . . . 100 ?chaptername? 1 Introduction (Français) 3 1.2.2 Objectifs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Résumé de nos contributions . . . . . . . . . . . . . . . . . . 4 1.4 Organisation du manuscrit . . . . . . . . . . . . . . . . . . . . 6

  Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Motivations and objectives . . . . . . . . . . . . . . . . . . . . 10 2.2.1 Motivation from Materials Sciences . . . . . . . . . . . . . . . 10

	1.4. Organisation du manuscrit
	?chaptername? 2
	Introduction
	Contents
	2.1

2.2.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Summary of our contributions . . . . . . . . . . . . . . . . . . 12 2.4 Organisation of the manuscript . . . . . . . . . . . . . . . . . 13

  Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

	Contents
	3.1 Super-resolution definition . . . . . . . . . . . . . . . . . . . . 15
	3.2 Super-resolution for materials data . . . . . . . . . . . . . . . 18
	3.2.1 Detecting local HR and LR images via Scale-invariant feature
	transform (SIFT) algorithm . . . . . . . . . . . . . . . . . . .	19
	3.2.2 Modeling contrast change parameters . . . . . . . . . . . . .	21
	3.2.3 Modeling super-resolution operator . . . . . . . . . . . . . . .	22
	3.3 State-of-the-art of Super-resolution method . . . . . . . . . 24
	3.4

  Thus compared with the deconvolution layer, ESPCN is capable of super-resolving LR to HR images with low computational cost at the time of training. More recently, inspired by recent work on Generative Adversarial Neural Networks (GANs) [35], C. Ledig et al. proposed a SR scheme with outstanding results in [56], called SRGAN. A GAN is a generative model in which two networks compete each other. The first network is the generator G, which generates a sample. While its adversary, the discriminator network D tries to detect if a sample is real or if it is the result of the generator.

  . In example-based approaches, the deep learning for SISR allows end-to-end reconstruction of HR image with successful results such as SRCNN, ESPCN, and SRGAN. However, these methods still have some limitations, such as high computation time, large database requirements, and convergence uncertainty. Besides that in this thesis, we focus on SR guided by high-resolution local sub-images of 3D materials data. This means that the training data is taken from the guided HR and LR part pair. Thus, the database is not large enough to train the network. Moreover we do not know the corruption downsampling operator when we only know the HR and LR image. This is also a great challenge for deep learning approaches. So in the first research of material data, we start to learn SR without deep learning approaches. The example-based approaches have been selected for research in this thesis, especially the SR approach based on joint dictionaries and sparse representations of image patches. The next chapter details some super-resolution approaches with Gaussian mixture distribution which based on patch-based approach. Then, we propose a new SR method with a generalized Gaussian mixture model. This method addresses all the requirements of superresolution processing for materials data, such as estimating the affine contrast change function C and the unknown super-resolution operator A. In particular, this method does not need to estimate the parameters of C, A and can still achieve good results.

	?chaptername? 4
	GMM and GGMM for
	Super-resolution

Contents 4.1 State of the art of patch-based approach . . . . . . . . . . . 30 4.1.1 Super-resolution via Joint Mixture Models based on MMSE [75] 30 4.1.2 Expected patch log-Likelihood (EPLL) method for superresolution [92] . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.1.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.2 Contribution: MMSE estimator with generalized Gaussian mixture model for super-resolution [69] . . . . . . . . . . . . 34 4.2.1 Parameter Estimation of Mixture Models . . . . . . . . . . . 35 4.2.2 MMSE Estimator for Generalized Gaussian Distributions . . 39 4.2.3 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . 41 4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

  4.2. Contribution: MMSE estimator with generalized Gaussian mixture model for super-resolution[START_REF] Nguyen | Image Super Resolution with a Joint Generalized Gaussian Mixture Model[END_REF] 41MMSE estimator T M M SE for the generalized Gaussian distribution is expressed as

  .2.26) 4.2. Contribution: MMSE estimator with generalized Gaussian mixture model for super-resolution [69] 43 Quality evaluation To evaluate the results, we use the Peak Signal to Noise Ratio (PSNR) value of two images X and Y . It measures the reconstruction quality of the fused image,

	P SN R(X, Y ) = 10 log 10 1	d 2 M SE(X, Y )	(4.2.27)

  .4. As a reference we also measure the PSNR of the nearest neighbor interpolation.

				FS		Diamonds
		d E-step M-step E-step M-step
	GMM	-717.91	0.07	718.13	0.07
	PCA-GMM, σ = 0.02	60 338.22 12.29 40 327.34 9.73 20 320.00 7.85	337.44 17.49 324.93 13.87 319.46 9.80
		60 337.29	4.15	337.42	4.16
	HDDC [11]	40 327.11	4.19	324.95	4.15
		20 320.03	4.20	319.07	4.15
	?tablename? 5.4: Average execution time (in seconds) of the E-step and M-step
	in the EM algorithm for estimating the parameters of the mixture models.
			d	FS	Diamonds
	Nearest neighbor -30.10		26.25
	GMM		-33.32		30.71
	PCA-GMM, σ = 0.02	60 33.38 40 33.36 20 33.25		30.83 30.75 30.17
			60 33.23		30.49
	HDDC [11]	40 33.24		30.29
			20 33.02		29.47
	?tablename? 5.3: PSNRs of the reconstructions of artificially downsampled 3D
	images using either nearest neighbor interpolation, GMM or PCA-GMM for different
	choices of d. The magnification factor is set to q = 2. As in the 2D case, PCA-
	GMM with small d produces results almost as good as GMM, but with a much lower
	dimensionality.

  Generative adversarial neural network . . . . . . . . . . . . . 90 6.2.2 SRGAN method [56] . . . . . . . . . . . . . . . . . . . . . . . 92 6.3 Contribution: Residual dense GAN (RDGAN) network for contrast change problem of super-resolution . . . . . . . . . 94 6.3.1 Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.3.2 Discriminator . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 6.3.3 Numerical results . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

	Contents
	6.1 Convolutional neural network (CNN) for SISR . . . . . . . 85
	6.1.1 Convolutional neural network (CNN) . . . . . . . . . . . . . .	86
	6.1.2 Super-resolution CNN method (SRCNN) [24, 25] . . . . . . .	88
	6.2 Generative Adversarial Neural Network (GAN) . . . . . . . 90
	6.2.1

  Quantitative results obtained on FS and Sic Diamonds images, the best results are in bold.

				Sic Diamond
	Quality evaluation PSNR Total variation PSNR Total variation
	ideal value	max	min	max	min
	Bicubic	32.01	0.0168	24.94	0.0299
	MMSE-GGMM	33.68	0.0152	27.22	0.0266
	RDGAN	33.76	0.0153	27.33	0.0261
	RDGAN-Contrast 33.71	0.0151	27.62	0.0259
	?tablename? 6.1:

https://www.math.u-bordeaux.fr/~jaujol/suprematim/

http://www.mummering.eu/index

https://www.psi.ch/en/sls/tomcat

https://www.psi.ch/en/sls

https://www.plafrim.fr/

https://github.com/johertrich/PCA_GMMs

https://github.com/johertrich/Inertial-Stochastic-PALM

, and friends, whom I would like to thank warmly.

First of all, I would like to express my deepest gratitude to my part of this jury as examiner. To all of them, thanks for taking the time to read this manuscript and for all the evaluation and the constructive comments.

I wish also to express my thanks to the members of my follow-up committee ("comite de suivi individuel"), Professor Jean-Pierre DA COSTA and Professor Yves COUDIERE, for several helpful comments concerning on the progression of my Ph.D. and the working environment during the preparation of this thesis.

In particular, I would like to give my special regards to University of Bordeaux and Agence Nationale de la Recherche (ANR) for providing financial support for my thesis to carry out the SUPREMATIM project. I was very fortunate to be involved in this project during my PhD. In this project, I am very happy to cooperate with Professor Gabriele STEIDL and doctor Johannes HERTRICH from the Technical University of Berlin (TU Berlin) and Professor Claudia REDENBACH from the Technical University of Kaiserslautern-Landau

?chaptername? 5. Dimensionality reduction ?figurename? 5.1: Plot of ordered eigenvalues of S k (left) and differences between eigenvalues (right).

Contribution: Dimensionality-Reduced Gaussian

Distribution: PCA-GMM Model [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] The main purpose of this section is to propose our contribution in the subject of dimensionality reduction data. The first contribution is a dimensionality reduction method that combines the PCA model with the Gaussian mixture model, called PCA-GMM model. This contribution was published in the Inverse Problems in Imaging journal [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] in 2022.

PCA-Gaussian mixture model

A first idea would be to couple the GMM and the PCA model in an additive way and to minimize for data samples X = {x 1 , ..., x N } in R n the function

Σ k ∈ SPD(d), k = 1, . . . , K, where △ K := w = (w k ) K k=1 ∈ R K ≥0 : K k=1 w k = 1 and X low := {U T y 1 , . . . , U T y N }, y i = x i -x.

Contribution: Dimensionality-Reduced Gaussian Distribution:

PCA-GMM Model [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] 55

It is important that the negative log-likelihood function L acts with respect to θ only on the lower dimensional space R d . The function F can be rewritten as

(5.3.3)

However, knowing that the samples were taken from K different Gaussian distributions it makes more sense to reduce the dimension individually for each distribution. Based on the reformulation (5.3.3) and using the notation

, we propose to minimize the following PCA-GMM model:

(5.3.5)

Clearly, if U k = U and b k = x for all k = 1, . . . , K, we get back to model (5.3.1).

The next lemma shows that our PCA-GMM model can be rewritten as a GMM model whose parameters incorporate those of the PCA.

and let f be the Gaussian density function (4.1.6). Then the following relation holds true:

where

(5.3.9)

?proofname? 1. First of all, we verify that the matrices Σ are well defined, i.e. that 1

?chaptername? 5. Dimensionality reduction By Lemma 5.3.1, the negative objective function in (5.3.37) is given by

(5.3.50)

In the following, we use const as a generic constant which has values independent of µ, Σ, U and b. The linear trace operator tr : R d,d → R fulfills x T Ay = tr(Axy T ) and in particular x T U U T x = tr(U T xx T U ). Using this property we obtain

= tr(I).

(5.3.51)

Thus, the only term in (5.3.48) which depends on b and U is G 1 . Further, minimizing G 1 is equivalent to minimizing

For fixed U , we can minimize g 1 with respect to b by setting the gradient to 0. Since g 1 is convex in b this is equivalent for being a global minimizer. This yields

In particular, b = 1 w m is a global minimizer of g 1 resp. G 1 , and it is independent of U . Using this, we get

Minimizing G 1 with respect to U for b = 1 w m is equivalent to minimizing

(5.3.52)

Further we have log 1 w U T SU = log(|U T SU |) + const. Thus, by combining the above computations, we get that minimizing (5.3.48) with respect to U is equivalent 5.3. Contribution: Dimensionality-Reduced Gaussian Distribution: PCA-GMM Model [START_REF] Hertrich | PCA Reduced Gaussian Mixture Models with Applications in Superresolution[END_REF] 63

to minimizing

ii) Now consider the case, where σ is learned. Again by (5.3.7), the maximizer in (5.3.37) with respect to σ is given by the maximizer of

By setting the derivative to zero, one obtains, that

Then the function in (5.3.49) modifies to

Now the monotonicity of the logarithm implies that minimizing G 1 is again equivalent to minimizing g 1 . Hence, as in case i) we get b = 1 w m is a global minimizer of g 1 resp. G 1 , and it is independent of U . Using this, we obtain

By (5.3.54), minimizing G 1 with respect to U for b = 1 w m is equivalent to minimizing

such that minimizing (5.3.48) with respect to U is equivalent to minimizing

) ∈ arg min

Note that the large data set X is involved in the computation of m k and C k , but it does not influence the computational time for minimizing the G k , k = 1, . . . , K. Indeed, the E-Step of Algorithm 2 will be the most time consuming one.

PALM for Minimizing G

To minimize G in (5.3.41) we propose to use the Proximal alternating linearized minimization (PALM) [START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF], resp. its accelerated version iPALM [START_REF] Pock | Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems[END_REF], where the 'i' stands for inertial. As a special case the PALM algorithm can be applied to functions of the form

where

where P(R d ) denotes the power set of R d .

Starting with an arbitrary x (0) PALM performs the iterations r) ) .

(5.3.64)

Remark 7. In fact, this special case of the PALM algorithm with one variable U for the optimization problem (5.3.63) is just the classical Forward-Backward algorithm.

Contribution: Dimensionality-Reduced Gaussian Distribution: PCA-GMM Model [41] 65

The Forward-Backwatd algorithm is also supposed to converge in the non-convex case as soon as the function satisfies the Polyak-Lojasiewics condition (see Theorem 5.1 in [START_REF] Attouch | Convergence of descent methods for semialgebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods[END_REF]). Thus, we can use the Forward-Backwatd algorithm with equation (5.3.64) as iterations of the algorithm. In this thesis, to get a faster scheme, we propose to use the iPALM algorithm to minimize G, which is detailed in Algorithm 3.

Indeed, we have applied the iPALM algorithm in our numerical examples. However, although we observed convergence of the iterates numerically, we have not proved convergence theoretically so far. Alternatively, we could apply the PALM algorithm which is slightly slower. Note again, that the E-Step of the algorithm is the most time consuming one.

Algorithm 3 iPALM

Input: α (r) , β (r) initialization x (1) , x (0) for r = 1, 2, ... do until a convergence criterion is reached

end for

In the following, we give details on PALM for our setting. For our problem (5.3.40), we choose f (U ) := ι St(d,n) and

where

otherwise is a smooth cutoff function of the interval (-ρ, ρ) for some ρ > 0. Then, the iteration scheme reads as

where Π St(d,n) denotes the orthogonal projection onto the Stiefel manifold.

Remark 8. (Projection onto Stiefel manifolds) Concerning this orthogonal projection, it is well known [START_REF] Higham | Matrix nearness problems and applications[END_REF], that for a matrix A ∈ R n,d , the projection Π St(d,n) (A) is given by the orthonormal polar factor W from the polar decomposition

Further, this orthonormal polar factor can be computed by W = U V , where A = U ΣV is the singular value decomposition of A, see [START_REF] Higham | Matrix nearness problems and applications[END_REF]. The authors of [START_REF] Higham | Fast polar decomposition of an arbitrary matrix[END_REF] propose to use the so-called Schulz-iteration

with X 0 = A for computing the orthonormal polar factor of a full rank matrix A.

Unfortunately, the convergence of this iteration requires that ∥I -A T A∥ F < 1, which is usually not fulfilled in our case.

Note that for any r ∈ N, the matrix U (r) belongs to the Stiefel manifold, such that η(∥I d -U T U ∥ F ) = 1 in a neighborhood of U (r) . Thus, we can replace the gradient with respect to H by the gradient with respect to G in (5.3.69). Then the iteration scheme reads as

In particular, we do not need to choose the ρ explicitly within our algorithm.

To show convergence of the algorithm, we need the following two lemmas.

Lemma 5.3.2. Let H be defined by (5.3.68). Then the function ∇H is globally Lipschitz continuous.

?proofname? The function H is twice continuously differentiable and zero outside of a compact set. Hence the second order derivative is bounded and ∇ U H(•, b) is globally Lipschitz continuous.

Further, let us recall the notation of Kurdyka-Łojasiewicz functions. For δ ∈ (0, ∞], we denote by Φ δ the set of all concave continuous functions ϕ : [0, δ) → R ≥0 which fulfill the following properties:

1. ϕ(0) = 0.

2. ϕ is continuously differentiable on (0, δ).

3. For all s ∈ (0, δ) it holds ϕ ′ (s) > 0.

For a proper and lower semicontinuous function γ : R d → (-∞, +∞] denote by ∂γ the subdifferntial of γ.

Definition 1 (Kurdyka-Łojasiewicz property). A proper, lower semicontinuous function γ : R d → (-∞, +∞] has the Kurdyka-Łojasieweicz (KL) property at ū ∈ dom ∂γ = {u ∈ R d : ∂γ ̸ = ∅} if there exist δ ∈ (0, ∞], a neighborhood U of ū and a function ϕ ∈ Φ δ , such that for all

Contribution: Dimensionality-Reduced Gaussian Distribution: PCA-GMM Model [41] 67

We say that γ is a KL function, if it satisfies the KL property in each point u ∈ dom ∂γ.

Lemma 5.3.3. The function H defined in (5.3.68) is a KL function.

?proofname? The functions G and η are sums, products, quotients and concatenations of real analytic functions. Thus, also H is a real analytic function. This implies that it is a KL function, see [5,Remark 5] and [START_REF] Łojasiewicz | Une propriété topologique des sous-ensembles analytiques réels[END_REF][START_REF] Łojasiewicz | Sur la géométrie semi-et sous-analytique[END_REF].

The following theorem follows directly from [9, Lemma 3, Theorem 1].

Theorem 5.3.4 (Convergence of PALM). Let F : R d → (-∞, ∞] be given by (5.3.63) and let ∇H be globally L-Lipschitz continuous. Let (x (r) ) r be the sequence generated by PALM, where the step size parameters fulfill τ (r) ≥ γL for some γ > 1. Then, for η := (γ -1)L, the sequence (F (x (r) )) r is nonincreasing and

). If F is in addition a KL function and the sequence (x (r) ) r is bounded, then it converges to a critical point of F . By Lemma 5.3.2 and 5.3.3 and the fact that G coincides with H in a neighborhood of the Stiefel manifold we obtain the following corollary.

Corollary 2. Let (U (r) ) r be generated by (5.3.70) with τ (r) ≥ γL, where L is the Lipschitz constant of ∇H and γ > 1. Consider the sequence generated by PALM with (5.3.70). Then, the sequence (G(U (r) )) r is monotone decreasing and the sequence (U (r) ) r converges to a critical point of G.

PCA-GMM model with application in super-resolution

In this subsection, we adapt the super-resolution method proposed by Sandeep and Jacob [START_REF] Sandeep | Single image super-resolution using a joint GMM method[END_REF] to our PCA-GMM model. The method follows 3 steps as described in subsection 4.1.1. The highlight of our method is the use of PCA-GMM model to learn the parameters of the mixture model instead of directly learning the original high-dimensional data.

For given low-resolution patches xL,i ∈ R τ 2 of an image and their high-resolution counterparts xH,i ∈ R q 2 τ 2 , q ∈ N, q ≥ 2, i = 1, ..., N , we learn a PCA-GMM model based on the data x i = xH,i xL,i ∈ R n , n = (q 2 + 1)τ 2 by Algorithm 2. This provides us with parameters {U, b, w, µ, Σ} of the reduced d-dimensional GMM. Using these parameters, we compute the parameters of the corresponding highdimensional mixture model w k , μk , Σk , k = 1, . . . , K, where µ k and Σ k are defined as in (5.3.8) and (5.3.9). In the following, we use the notations μk = μH,k μL,k and all k = 1, ..., K. At each k mixture component of the mixture model, the parameters are given by: ?proofname? Lemma 3.1, p. 5 [START_REF] Hult | Multivariate extremes, aggregation and dependence in elliptical distributions[END_REF].

The PCA-GGMM model can be used for super-resolution based on MMSE estimator approach as in Section 4.2.2. We can apply the MMSE estimator in equation (4.2.22) presented in Section 4.2.2 to the generalized Gaussian mixture model with parameters θ k of the high dimensional data. Finally, the highresolution image is reconstructed by applying the reconstruction of high-resolution image step by patch averaging as in 3, Subection 4.1.1.

Numerical results

In this section, we demonstrate the performance of PCA-GMM model and PCA-GGMM model with applications in super-resolution by two-and three-dimensional examples, where we mainly focus on material data which introduced in Chapter 3.

Initialization of the EM algorithms. Since the negative log-likelihood function is non-convex and admits many critical points, EM algorithms for GMMs are very sensitive with respect to the initialization. For example this can be seen by considering the case that θ , k, l = 1, ..., K. The same effect appears for PCA-GMMs, PCA-GGMMs and HDDC. Consequently the initialization of the EM algorithms is of great importance. For our numerical examples, we initialize the GMMs as follows. We set w

K , for k = 1, ..., K. For initializing the means, we choose randomly K distinct data points µ 1 , ..., µ K from our training data x H,1 , ..., x H,N . Finally, we choose for each k = 1, ..., K the M points x L,1 , ..., x L,M from x H,1 , ..., x H,N which are the closest ones to µ k and initialize the covariances by

. The number M is chosen according to the dimension n of the data. In our examples, we use M = 2n.

We initialize the PCA-GMMs, PCA-GGMMs and HDDC by taking the initialization for GMMs, running one E-Step from the EM algorithm for GMMs followed by the M-step of the PCA-GMMs, PCA-GGMMs or HDDC, respectively. change challenge ?figurename? 6.1: Scheme of max pooling • Average pooling: It calculates the average value in each neighborhood and sends to the output array (Figure 6.2).

?figurename? 6.2: Scheme of average pooling Although a lot of information is lost in the pooling layer, it reduces complexity, improves efficiency and reduces the risk of overfitting.

Fully-connected (FC) layer Fully connected layers connect every neuron in one layer to every neuron in another layer. This layer performs the classification task based on the features extracted by the previous layers and their different filters. While convolutional and pooling layers tend to use ReLU functions, FC layers usually use the softmax activation function to calculate the probability distribution from 0 to 1.

6.1.2 Super-resolution CNN method (SRCNN) [START_REF] Dong | Learning a deep convolutional network for image super-resolution[END_REF][START_REF] Dong | Image super-resolution using deep convolutional networks[END_REF] In this section, we mainly discuss the network architecture of super-resolution CNN (SRCNN), which was proposed by C.Dong et al. [START_REF] Dong | Learning a deep convolutional network for image super-resolution[END_REF][START_REF] Dong | Image super-resolution using deep convolutional networks[END_REF]. SRCNN has three main parts: patch extraction and representation, non-linear mapping, and reconstruction, as shown in Figure 6.3. The input of this network is a bicubic-interpolated image, which is upscaled from the low-resolution image and has the same resolution of the high-resolution image. We denote the bicubic-interpolated image as X L and ?bibname?

Abstract:

Recent developments in imaging techniques and computational analysis deeply modify the way materials sciences. The materials image has moved from microscale resolution to true nanoscale to analysis the defects and details at the interfaces of the materials. Thus, this thesis deals with the super-resolution (SR) problem in order to reconstruct the materials images in the high-resolution for instance the nanoscale resolution. To reach this goal, this thesis explores several SR approaches for materials images.

Single Image Super-Resolution (SISR) algorithms based on patch-based have been noticed and widely used over the past decade. Recently, Generalized Gaussian Mixture Models (GGMMs) have shown to be a suitable tool for many image processing problems due to the flexible shape parameter. In the first place, we propose using a joint GGMM learned from concatenated vectors of high and low resolution training patches to do super-resolution image based on the minimum mean square error (MMSE) method.

Unfortunately, the dimension of the concatenated samples is too high for the learning of the mixture models such as Gaussian mixture model and GGMM. Then we propose these two models Gaussian mixture model (GMM) and GGMM with a reduction of the dimensionality of the data in each component of the model by principal component analysis. These models are called to as PCA-GMM and PCA-GGMM, respectively. To learn the (low dimensional) parameters of the mixture model we propose two different EM algorithms whose M-step requires the solution of constrained optimization problems. Then we apply our PCA-GMM and PCA-GGMM for the super-resolution of 2D and 3D material images based on the MMSE method for the GGMM model.

In addition, the study of the material data becomes difficult because the HR and LR images have a different contrast. In our last contribution, we study a deep learning approach considering the problem of contrast change in material images. Indeed, we propose a generative adversarial network (GAN) within two generators, each responding to a different task. The first generator deals with the problem of contrast changes, and the second one focuses on the reconstruction of the high frequencies of the SR.

All the proposed methods lead to convincing results, both quantitative and visual. Especially the numerical results of the dimensionality reduction methods confirm a moderate influence of the size reduction on the overall SR results.